Yahoo’s new interface

[Sandy & Shirley]

Yahoo Finance changed the way they present their data about a month ago.

I’m looking for another computer geek to help me with a problem that caused.

The ability to ask for the current prices on a list of symbols, copy the screen display, paste it inside an excel spreadsheet, and then extract the data is gone. The copy no longer contains the symbols for each requested stock or fund.

There are many places on the web where you can ask what symbols are performing the best, “TODAY”! I also wanted to know what were the best symbols a month ago, 2 months ago, etc. So that I could look for trends.


So, as a computer geek, I wrote software that used yahoo finance to get 5 years of historical market data for a list of symbols starting on any date that I wanted to see. What did the market look like on Jan 1 2017?

I can still do this on-line with a web browser, but it no longer works inside my computer program.

Does anyone know of another source for historical data?

[Lsbcal]

One easy solution to your problem. Find better investing techniques. Sorry, I know this is not what your were looking for.

Have you benchmarked your technique versus other easy to manage ones?

[Birdie Num Nums]

As a related aside, virtually every time I visit Yahoo News I get this fake "critical Mozilla Firefox patch update." I immediately close that pop-up window. And the other day I also got a fake Zeus malware warning ("do not close window or shut down your computer" warning, but call this phone number for a "fix"). Google that!

[Blue Collar Guy]

i find yahoo finance to be inaccurate. they havent updated the adjusted closing price in a long time

[Sandy & Shirley]

Quote:

Originally Posted by Lsbcal One easy solution to your problem. Find better investing techniques. Sorry, I know this is not what your were looking for. Have you benchmarked your technique versus other easy to manage ones?

Yes I have benchmarked my “investing technique”. I have been using it a little over 3 years now and over any given 12 month time frame my results have been between 6% to 10% higher than the yearly average ROR of the 3 major market indexes. I do realize that those indexes do not include things like dividends, but that only accounts for about 3% of my advantage. So, basically I am consistently doing better than the market.


Do I make mistakes, definitely, but I cash in on trends far more than losing. I just need accurate historical data to feed my software.

[GrayHare]

Is it historical price data you want? Investopedia has it, such as this link for AT&T:

http://www.investopedia.com/markets/stocks/t/

[Lsbcal]

Quote:

Originally Posted by Sandy & Shirley Yes I have benchmarked my “investing technique”. I have been using it a little over 3 years now and over any given 12 month time frame my results have been between 6% to 10% higher than the yearly average ROR of the 3 major market indexes. I do realize that those indexes do not include things like dividends, but that only accounts for about 3% of my advantage. So, basically I am consistently doing better than the market. Do I make mistakes, definitely, but I cash in on trends far more than losing. I just need accurate historical data to feed my software.

How about looking into the data source Yahoo uses? You would probably have to pay. Once I looked at their site and they had various subscriptions. I think Yahoo has them listed in a footnote to some data, maybe in the historical data section.

You could call up Schwab. They have some of their own software and must have data too.

[Sandy & Shirley]

I found the answer. The CSV data is no longer accessible for computer program screen scraping, but the standard HTML format can still be used.

This post is for the total geeks who are trying to screen scrape historical data. If you are not a total geek you are probably scratching your head already and asking what the heck is screen scraping!

Yahoo now uses time_t to represent the starting and ending dates for historical data. Time_t is the number of seconds since 1/1/1970 represented as a 32 bit signed integer. It will reach its max on January 19, 2038, so we still have time to use it.

The yahoo finance URL for getting the 37 monthly adjusted closing prices for DNP from May 1, 2014 to Jun 1, 2017 is:

https://finance.yahoo.com/quote/DNP/history?period1=1398916800&period2=1496289600&inte rval=1mo&filter=history&frequency=1mo


It uses time_t values for the date range and sets the interval and frequency for monthly data.

OK, if you are not a total geek you will now get totally lost.

Scan the HTML code of the returned page for “Currency in USD”, then search for each table row TR in the first TABLE that follows that string.

Remember the content of the first column in each row as the date. If there is a TD 6th column /TD, then this is a historical adjusted closing price for that date. If there is not a 6th column the row is only telling you the dividend or split information.

If you are a total geek and are interested in the code that I wrote, send me a PM and we can talk more about it. If you are not a total geek you probably didn't read to the end of this post anyway!

[Lsbcal]

Quote:

Originally Posted by Sandy & Shirley I found the answer. The CSV data is no longer accessible for computer program screen scraping, but the standard HTML format can still be used. ....

I am a little confused. I use Yahoo historical to get monthly performance data. When I download a series it comes across as a .csv file. The numbers seem to be consistent recently. I checked that the monthly returns for VINEX are the same as my past data for the last 7 years.

So if by "screen scraping" you mean taking the "Adj. Close" prices to compute the monthly returns, then that still seems to be fine. Note that the numbers are adjusted for new dividends. So all the numbers in a series are backward adjusted on the new dividend month. But the differentials (to compute the monthly return) should remain the same.

[Sandy & Shirley]

What I’m talking about, geek speak, is the difference between using a browser like IE or FF to look at the data with your eyes, and in this case click on a link to download the data and get a popup screen asking how you want the data sent back to you. That browser has been keeping special protocol data, like cookies (geek speak), which it can pass back to yahoo so that they know that the browser is authorized to receive the requested CSV file.

If I understand you, you do this one symbol at a time, open the returned file in excel, and then do your analysis.

The software I wrote is following 355 symbols at the same time, based on the list of Fidelity no transaction fee funds. At the end of the trading day, maybe 7 PM after all of the funds have posted their closing prices, I click a single link on my desktop. A “command prompt” window opens up and begins to display today’s data. It take 2 to 3 seconds per symbol and finishes in about 10 minutes at which time the widow makes a series of quick beeping sounds, waits 10 seconds just in case I want to look at it for some reason, then closes on its own.

I then run my own “Market View” software that can look at a snapshot, calculate the monthly, quarterly, and annual RORs for each symbol and display which symbols did the best for each of those time periods. It also allows me to create custom “views” of the market where I can ask which symbols did the best when looking at multiple RORs at the same time, show me the best 1 year returns that are still doing well over the last 3 months.

My snapshot program can also be run for historical dates, what the market looked like on March 31, 2017. I keep monthly historical snapshots for the past 5 years. All of the data that I can find on line lets me look at what the best investments are “today”. I can analyze the latest snapshot and also get the same information, but I can then ask what was the best investment a month ago, the month before that, or last year, etc.

All of this is only possible because I can look at 355 symbols on any historical date in about 10 minutes or less. I have 5 years of monthly snapshots saved on my disk. My custom market view software loads all of this data and computes all of the RORs and Rankings in less than 5 seconds so that I can then view the data instantly in multiple formats.

This is only possible because of the snapshot process that can “look at” 355 web pages in 10 minutes, extract (screen scrape) the data I want and save it on my disk.

That program used to be able to ask for the CSV data directly from yahoo, but they changed their protocol for requesting that data so all of my personal market analysis software came to a screeching halt. I now have to analyze the background HTML to extract that data. If you want to see what I am talking about, display any historical view page on your browser, right click on the page and select “view page source”. That is what the HTML looks like that displayed the humanly readable page on your browser. That is what a screen scraping process looks at one character at a time, to find and save the data you want.

[Lsbcal]

OK, thanks for the overview. Didn't realize you were looking at such a large set of funds. My needs are much more modest and for equities I only need about 10 fund monthly updates.