CLI Data Gem Project: True Crime Documentary "Database"

At last, I am complete with my project.  It took exactly two weeks.


The name of my project is True Crime Documentary Database. It offers a list film and television documentary film titles in the True Crime genre. 

Screen Shot 2018-06-23 at 7.03.36 PM.png

(Note: Although it indeed is not a database, its functions point in the direction of a database. The term "database" is used here loosely.)  

I used the Nokogiri gem to scrape the site  

Screen Shot 2018-06-23 at 7.20.39 PM.png

I created an application that scrapes the category names, the URLs, and the documentary attributes that exist within each URL path.  I had to scrape these link (image below).

Screen Shot 2018-06-23 at 7.22.56 PM.png

The program should deliver a menu of options: to browse the documentaries by category or to see a full listing of documentary titles.  

Screen Shot 2018-06-23 at 7.03.47 PM.png

Selecting category delivers a list of documentaries displayed with their title, year, category, synopsis, and full synopsis URL.  

Screen Shot 2018-06-23 at 7.04.09 PM.png

Otherwise, if list all documentaries is chosen, then the user is taken to a submenu from which to select a title.  Similarly, the selection displays the documentary’s details.

Screen Shot 2018-06-23 at 7.04.27 PM.png
Screen Shot 2018-06-23 at 7.04.41 PM.png

Building The Project

Now, based on my initial review of the HTML I felt assured that I could achieve this project based on what I had learned in the previous labs leading up to my CLI data gem.  What I hadn’t given better attention to was the layout of the site and how it delivered the data.  The categories of a film are front and center, while the documentaries generate (no HTML to scrape) in many places, except inside of category link paths in the right sidebar.  Sadly, I hadn’t noticed this quirk until long after getting approval and having coded my scraper class. (With respect to object orientation, this presents some coding challenges. It would be hard to maintain the has a/has  many object relationship that I planned.)  


In the scraper class is when I realized I might run into trouble when I’d have to connect these objects in the CLI controller class.  Sure enough, that difficulty arose. I worked hard to refactor the entire project from start to finish and ended up with a better product.

Something I encountered was that the number of links I decided to scrape tacked on at least 15 to 20 seconds of load time at the start of the program. Coincidentally, in one of my recent study groups, the instructor mentioned how it might sometimes be a better option to have the data load (scrape) upon selection by the user. I thought it was an excellent idea!  However, given how far I had come and that I was already in the middle of rebuilding my program from scratch (I created a branch and merged later), it would have proven difficult timewise.  So again, the layout and coding of the site presented a hurdle that made any new idea unfeasible.

Overall, I believe that I have met the requirements of the project by demonstrating that I have a good grasp of object orientation and working confidence in coding in Ruby.  

Some Afterthoughts

  •  Simple ideas are still complicated when it comes time to code them!
  •  What you think is simple may not be simple enough.
  • *Projects fan the flames of enduring appreciation for Google search! :)
  • *This project was, *hard*! However, it I love it! 
  • *Coding is fun. Learning to code is fun and uncannily cathartic. O_O

Bottom line: The CLI data gem was a challenging project, while also gratifying!

Access the repository here.

CLI Data Gem Project Complete...mostly

I'll explain how it works later in the week.  But for now I'm just celebrating having made it through the toughest part, which was building it and getting it to work.  It took me a week.

Screen Shot 2018-06-20 at 1.48.20 AM.png
Screen Shot 2018-06-20 at 1.48.49 AM.png
Screen Shot 2018-06-20 at 1.49.29 AM.png

A Way to Learn As a Flatiron School Student

I began my journey as a FS Web Development student in the first week of April. I am a quarter of the way through the curriculum and would like to share some patterns that I have noticed in my learning in hopes that it will be of some help to any other beginning FS student.

What I have learned about my learning:

  • Get comfortable with being uncomfortable with broken code.  In fact, get comfortable with breaking and redoing or refactoring code to higher efficiency. In a phrase, think in drafts. Writing code is not unlike writing a paper that is too wordy, uses overused terms, or is poorly formatted.  Working code is not necessarily "good" code.  All of this is part of the ongoing process of programming.  Think: Less is more. Simplest or abstract code is the good code.  Although, I grant that for beginners that the "efficient" is relative to our experience and what little we know.  So be patient with yourself.
  • Read test specs carefully and plan accordingly.  As great as it is to have pre-written test specs, don't make the test specs a crutch.  Remember this is a learning environment intended to simulate and teach; that is, to train as you fight, figuratively speaking. This is to say, you need to learn the way a programmer is expected to program in the real world or learn  by doing as programmers do. However, is still a teaching environment and not the real world.  Therefore, plan and think, as opposed to slipping into autopilot--because it is possible up until a big challenge comes along and appears wholly unfamiliar on top of whatever labs are supposed to be challenging all their own. Don't get me wrong! These lessons and labs are scaffolded to comprehensively challenge you.  But this does not rule out the personal responsibility to review, take some notes, and make connections to prior knowledge; assimilating knowledge, while consistently taking in entirely new experiences.
  • Strategize, strategize, strategize!  Object-oriented ruby, for example, will begin to introduce you to more realistic interaction with code, where you will code in one file while having to pause work on that file to accomplish some working code in another to enable the prior file to work.  The implied task is that you will need to understand what to code in some necessary order.  This makes the previous steps crucial.
  • Think about your thinking and approach. For example, when you make an error, and whether you are able to resolve it quickly or not, take a moment to understand how you got to that point, whether your code is passing or failing the tests specs.  In terms of a failing spec, think:  What am I missing in this method/class/lab?  What does the test spec say?  Does my code address every facet of what the spec requires? (On the latter, all but a missing piece of code could be causing an error to be thrown, demonstrating while you have to think both with and beyond the given test specs! Stated inversely, all your code is set to work if one piece of code weren't missing.)  So you kind of have to be a Sherlock who must really observe your code to embrace the possible problem that just isn't popping out at you despite having run yourself through several loops to figure out why your code is not working.  And in terms of a passing spec, ask yourself:  How did I accomplish this?  Do I know and understand what I did to achieve this success?  Implied task:  Take time to evaluate your learning, even if you are a high-speed FS student with previous experience.
  • Evaluate and document (summarize in your own words) what is being asked of you to  accomplish a particular specification(s). Although has built-in tests for you, you still have to do the work of thinking. As you progress through the curriculum the tests will become less and less specific. There will be gaps in the logic of the test specs that require you to fill in the blanks for how you get from one end of a problem to its expected outcome or output. 
  • Remember that computer programming's technical and essential purpose:  To solve real-world problems.  You should be excited by the presentation of a problem to solve, as opposed to feeling bothered by problems.  So to further break a broken record, learn to love problems so that you can get to the place to appreciate how your work as a programmer can improve people's daily lives, because that is what it's all about at the end of the day.

Dig deep. Think. Code. Evaluate. Adapt. Keep going. :)

Originally published at