Installing Theano on Windows 8.1

So I tried using Caffe for some natural language processing and found it wasn’t well suited. While it’s great for image processing with convolution networks (CNN), it would take quite a bit of effort to add looping output to make a recurrent (RNN). Recurrent Neural networks are helpful in natural language processing. For instance, if we’re going to pick the next word in a sentence, it’s important to know what words came before it.

Keras is the other major python based neural net tool and to use it, you need matrix library like Theano or Tensorflow. I chose Theano since Tensorflow doesn’t have good windows support at the moment.

The main issue I ran into while installing Theano a crash I was running after installation. The Windows guide can be found here. The dependencies all installed more or less without a hitch. However, I foolishly thought my gcc installation was sufficient, and eventually ran into problems the first time I tried:

conda install theano

I hit a wall. Unfortunately, I didn’t catch what that first error was. The bottom line is I eventually realized it was missing compiler error, and I thought I solved it with

conda install mingw

This doesn’t work! I was able to get Theano to import in python after I added the .theanorc file to my %USERPROFILE%, but I was still running into a python crash specifically at this line in the gpu test:

f = function([], tensor.exp(x))

Python crashed and burned and I figured it had to do with mingw not using a 64-bit binary or dll. No evidence though. Anyway, this fixed it for me:

conda uninstall mingw
conda install m2w64-toolchain

If I had followed the directions closer, I might have saved quite a bit of time, but I just assumed GCC was fine.

Never assume your toolchain is right when you’ve got x86_64 flags in the mix until you’re compiling.

The State of Bitcoin in 2017

So I recently was corresponding with a friend about Bitcoin. Since I haven’t mentioned anything about my interest in cryptocurrency, here’s the summary I sent them, reposted with permission

~~~~~~~~~~~~~~~~~~~~~~~~~~

Totally in agreement that it’s a radical departure from previous asset transfer mechanisms.  Bitcoin is the first generation of tech that enables uncensorable payments in a state-free currency. Value transfer is the primary idea, but there are lots of other applications for a distributed ledger like notarization. A multiparty transaction can include a message like a contract that is an unalterable, timestamped proof of signatories. Anything that requires proof of timestamped signatures is a possible application (securities, insurance, rental agreements, ect).

The most important and infamous bitcoin personality worth researching a little is the inventor, Satoshi Nakamoto. One of the first developers, Mike Hearn, reported in 2013 that Satoshi said “the project was in good hands and he had moved on to other things”, and no one has heard from him since. He likely mined a significant amount and many of the first addresses that contain thousands of bitcoin that haven’t ever been touched. They’re continuously monitored and if they’re ever accessed it would likely cause a major correction to the price. It’s rumored that Satoshi may be an alias for a group of people, possibly even deep state security researchers. If it’s ever revealed proof positive that the NSA can trivially decrypt the RSA algorithm which the tech is based on, every wallet is essentially compromised, i.e. the tech is worthless.
The first big exchange was Mt. Gox run by Mark Karpelès. The exchange was hacked and the price tanked. This was not be the last time this happened and it will happen again. The point of that story is that if you don’t have the coins in your own wallet, you’re trusting the other party completely. The big players right now are still the exchanges and their founders since the fiat value is what most people are speculating and trading on. Brian Armstrong started Coinbase which is US based and is the most reputable.
 
The biggest player on the development side is a company called Blockstream and a wackjob dev, Luke Jr. They controversially have been opposed to increasing the block size to lower fees and process more transaction. This has caused a lot of debate since their company has developed a protocol layer on top of Bitcoin called the “lightning network” to enable faster/lower fee transactions. In my opinion, turning the Bitcoin network into a ‘settlement layer’ for high value transactions isn’t the right solution and Satoshi originally described increasing the block size to handle more transactions so they’re acting more in the spirt of a traditional payment processing company.
 
This brings us to the latest hardfork of the code Bitcoin Cash that split off to start raising the block size. It created a separate blockchain but identical chain up to the fork. Bitcoin holders essentially control an equal amount of Bitcoin Cash tokens. The economics of the situation are somewhat unprecedented, since it created almost $7B overnight. The debate over which is the real bitcoin is somewhat moot. Whichever chain is longest by the virtue of the most hash power is likely to win, displacing the other. Definitely something to keep an eye on.
 
Of all the remaining Alt-coins, Ethereum is the most interesting, in my opinion. Started by a kid-genius named Vitalik Buterin. Instead of spending electrical power on inefficient hash computing cycles, it uses a different algorithm called proof-of-stake to create an uncensorable computer application. Combine it with distributed encrypted storage and this is a jump from an uncensored payment network, to an uncensorable information network. Imagine servers that can’t be shut down by fascist governments. It’s like the wild west of the internet circa ’96 or 97. The Ethereum foundation did just recently sign some agreement with a Russian bank to create some new token which I haven’t decided if I’m for or against. It’s certainly a step toward centralization.
 
There also is definitely an Alt-coin bubble happening right now. China just banned most Initial Coin Offerings (ICO’s) right after the SEC notable shut one down. Essentially they’re a fund raising tool where you trade bitcoin for newly created ‘coin X’. This sidesteps traditional VC raises completely. However the nature of virtual currencies being tentatively treated as securities caused some regulators to freak out when millions of fiat equivalents started being invested to unregistered entities on a whitepaper, song and a dance.
 
The easiest way to have a stake is buy coins. If you buy coins and want to move the off the exchange you’ll need a wallet like Electrum or Exodus. I don’t recommend the bitcoin-core wallet since it syncs the entire blockchain. Not terrible at 150GB but still a pain. Alternatively you could buy companies that accept and hold bitcoin.
I do recommend playing around with some crypto tools if you havent before. GPG4Win which includes the Kleopata key manager is kind of a pain, but it’s the only game in town besides the command line. My public key is here if you want to try send an encrypted message. http://www.raleighillgen.com/about/Keepass is also a good application to keep and generate secure keys.

Book Review — How To Win Friends and Influence People by Dale Carnegie

The classic HTWF&IP by Dale Carnegie was an interesting read. A lot of the suggestions are pretty timeless and obvious advice along the lines of, actually be interested in what the person you’re talking to is saying, offer compliments when appropriate, make sure to be sincere when doing so, do your best to be pleasant company. Like most self-improvement works, the author often relies on the wisdom of the ancients and you’ll come across quotes from western philosophers and a couple Lao Tzu references, along with a bevy of 19th & 20th century political and business giants.

There are a couple insightful psychological points when it comes to trying to change someone’s mind, like always try to get a couple ‘yeses’ from the person you’re trying to convince before your contentious point to increase the odds they’ll agree, and to make your presentation a little more dramatic since our minds are most accustomed to storytelling as a mode of understanding.

While a lot of the anecdotes feel dated and some of the social etiquette doesn’t seem to apply anymore in this, the age of post truth and the death of outrage, the book is still a good read if not a little long winded. Here are some of my favorite quotes along with the basic key points he makes throughout the book:

 

IN A NUSTHELL SIX WAYS TO MAKE PEOPLE LIKE YOU

  1. Become genuinely interested in other people.
  2. Smile.
  3. Remember that a person’s name is to that person the sweetest and most important sound in any language.
  4. Be a good listener. Encourage others to talk about themselves.
  5. Talk in terms of the other person’s interests.
  6. Make the other person feel important—and do it sincerely.

IN A NUTSHELL WIN PEOPLE TO YOUR WAY OF THINKING

  1. The only way to get the best of an argument is to avoid it.
  2. Show respect for the other person’s opinions. Never say, “You’re wrong.”
  3. If you are wrong, admit it quickly and emphatically.
  4. Begin in a friendly way.
  5. Get the other person saying “yes, yes” immediately.
  6. Let the other person do a great deal of the talking
  7. Let the other person feel that the idea is his or hers
  8. Try honestly to see things from the other person’s point of view.
  9. Be sympathetic with the other person’s ideas and desires.
  10. Appeal to the nobler motives.
  11. Dramatize your ideas.
  12. Throw down a challenge.

On understanding over condemnation:

Instead of condemning people, let’s try to understand them. Let’s try to figure out why they do what they do. That’s a lot more profitable and intriguing than criticism; and it breeds sympathy, tolerance and kindness. “To know all is to forgive all.”

Carnegie’s Tombstone:

Carnegie wanted to praise his assistants even on his tombstone. He wrote an epitaph for himself which read: “Here lies one who knew how to get around him men who were cleverer than himself.”

On the power of the mind:

Thought is supreme. Preserve a right mental attitude—the attitude of courage, frankness, and good cheer. To think rightly is to create.

On appeals to the logos:

Few people are logical. Most of us are prejudiced and biased. Most of us are blighted with preconceived notions, with jealousy, suspicion, fear, envy and pride. And most citizens don’t want to change their minds about their religion or their haircut or communism or their favorite movie star.

On treading softly:

The Chinese have a proverb pregnant with the age-old wisdom of the Orient: “He who treads softly goes far.”

On the importance of asking “why?”:

[…] before asking anyone to put out a fire or buy your product or contribute to your favorite charity, why not pause and close your eyes and try to think the whole thing through from another person’s point of view? Ask yourself: “Why should he or she want to do it?”

J. P. Morgan on motivation:

J. Pierpont Morgan observed, in one of his analytical interludes, that a person usually has two reasons for doing a thing: one that sounds good and a real one.

Book Review – Reinvent Yourself by James Altucher

This was a really enjoyable read. I was a little put off with the format at first, which is basically the author recounting various interviews he’s done for his podcast and a smattering of profiles on people he admires or has studied. After reading a few though, his analysis and commentary on the interviews became pretty compelling. James has started several businesses over his life and he makes it a point of studying the failures to see how he could improve from them. His interviews mostly summarize to do what you love, do it all the time, and do it to the best of your ability. Let everything else sort itself out. That, and always be kind. There’s no reason not to be.

Some of the quotes that resonated
On life in general:
I realized that I was an amateur. I had spent my life pretending to be something I wasn’t. I wanted to move beyond to that. TO:

  • Have humility. Learn from everyone you can. Even if it’s just one takeaway.
  • Be grateful for the many lessons you get, and realize that everything is a lesson.
  • Only be around people you love and who inspire you.
  • Life is a billion times smaller than the point of a needle. Don’t waste it doing things you were told to do. Do the things you love to do.
    Health is the most important thing, else your body today won’t let you enjoy tomorrow.
  • Every day, be creative. Creativity is a muscle. *You’re going to make mistakes, but 80% is always good enough. Keep learning the next thing.
  • Life will constantly hit you until you are senseless. Don’t forget these are lessons.

On always be improving:
You need to find well-being from within. And here is what it is: FREEDOM RELATIONSHIPS COMPETENCE Increase those every day and you will find well-being.

On Reinvention:
So reinvention is:

  • Defining freedom in different ways (reducing expectations, increasing sources of income so no one source controls you).
  • Improving relationships. Plus, minus, equal: Finding mentors to teach you. Finding the next generation to teach. Finding friends who build you up and challenge you. This is your “scene.” Everyone going through reinvention needs a scene.
  • Habits. It’s the 5×5 rule. You are not just the average of the five people around you. You’re the average of the five habits you do, the things you eat, the ideas you have, the content you consume, etc.

On the search for meaning:
“Find the thing you did where you lost all sense of time while you are doing it,” Chip told me. “Remember the equation from Victor Frankl’s Man’s Search for Meaning. “Despair = Suffering – Meaning. Find the things that bring you meaning. Suffering is always there in this world. But if you have meaning, you will have less despair. You will find your calling.”

On not stagnating:
The third day at the job, I got up and walked out. I didn’t clean out my office. I left my jacket there. I took the elevator down 40 stories. I walked out into the sun. And I never went back.

They called repeatedly. Even a year later the main guy was still calling. My life is better than ever. I never looked back. I left the building and walked to Grand Central. I took the train 80 miles. I watched the leaves turning from green to red along the way across the Hudson River.

On learning aggressively:
There are two ways to learn: passively and aggressively.
Passively is when you study your mistakes, read the history of what you are learning, network, find your “tribe,” find a mentor, etc. Aggressively is right when you are in the middle of it. You’re neck deep and the ball is coming at you: what do you do? Passively is in your head. Aggressively is noticing RIGHT NOW and taking action. In your head is important. But ACTION is what creates heroes.

On cooperation:
It turns out that evolution is not about individual selection. We only survive as well as we function in terms of a group. When we are a strong part of a group, when we help the group, and when we use the group’s resources to become better as individuals, then we survive and even thrive.

Book Review – The Compound Effect by Darren Hardy

I just finished “The Compound Effect” by Darren Hardy and it was a great reminder that success is the sum of a lot of small actions. The author insist that the reader needs to put their nose to the grindstone for an extended period of time before they should expect to see results, although as the title suggests, the results of effort can become more than just cumulative giving an exponential boost to what we’re trying to accomplish. Darren is the mentee of Jim Rohn, a major self-improvement guru I wasn’t familiar with, but may check out.

Here are some of the notable quotes I took away. On tracking key performance indicators:
To help you become aware of your choices, I want you to track every action that relates to the area of your life you want to improve.

On paying attention to who we spend our time with:
I’ve got a neighbor who’s a three-minute friend. For three minutes, we have a great chit-chat, but we wouldn’t mesh for three hours. I can hang out with an old high-school friend for three hours, but he’s not a three-day guy. And, then there are some people I can hang around for a few days, but wouldn’t go on an extended vacation with. Take a look at your relationships and make sure you’re not spending three hours with a three-minute person.

On asking others to help improve ourselves:
I have a serious challenge for you if you’re up for it. Want real feedback? Find people who care enough about you to be brutally honest with you. Ask them these questions: “How do I show up to you? What do you think my strengths are? In what areas do you think I can improve? Where do you think I sabotage myself? What’s one thing I can stop doing that would benefit me the most? What’s the one thing I should start doing?”

On paying attention to whether our environment is supportive:
The dream in your heart may be bigger than the environment in which you find yourself. Sometimes you have to get out of that environment to see that dream fulfilled. It’s like planting an oak sapling in a pot. Once it becomes rootbound, its growth is limited. It needs a great space to become a mighty oak. So do you.

I started a travel vlog!

So I’m currently in Chiang Mai, Thailand and loving it so far! I decided to start a travel vlog to share a little bit of the experience. I’m currently in Chiang Mai at least through March, but I considering going down to Ko Samui or Ko Phangen in April after I renew my visa for an extra 30 days. Apparently once burning season gets into full swing the air quality is sub par here in northern Thailand. That and I think I could use a little beach R&R.
After April, I’ll be at the end of my Thai single entry visa and am currently deciding between Vietnam, Myanmar and Indonesia for my next stop.

So here’s my vlog channel, Ral Around the World!
Ral Around the World

Compiling OpenCV on Windows 8.1

I’m setting up the latest OpenCV version from source since the Caffe dependency version installed doesn’t have some tools I want to add to my vision processing pipeline.

I started with grabbing the latest version from github
git clone https://github.com/opencv/opencv

Then I used the CMake GUI. Visual Studios 12 appears to be the latest buildchain supported
opencv-cmake

If you haven’t used the GUI version of CMake before, the most import feature you’ll use while setting everything up is the File – Delete Cache command. CMake needs the cache cleared after you fail a configuration attempt or a bad environment generation.

I was really lazy after grabbing the source and didn’t check any of the dependencies involved. CMake found my Cuda installation without any extra config, but there’s a python requirement for both versions 2.7 and 3.4 which I can’t get both to open correctly. %PYTHONHOME% is a serious PITA. Hitting the generate button a few times eventually yielded a successful config regardless.

Once you get the CMake to generate a configuration you can hit the Open Project button to open the project in Visual Studios.
From there, hit Build – Build Solution to start compiling.

Everything appeared able to compile with 6955 warnings and 1 error related to a python27_d.lib linker error that hopefully only broke the python interface.
error LNK1104: cannot open file 'python27_d.lib'

An optimization that’s probably worth is exploring is explicitly picking the CUDA architecture as I think it built it for all possible versions. My suspicion stems from these warnings:
warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
And the fact that the opencv directory is now over 8GB which seems on the large side.

Setting Up Caffe on Windows 8.1

It took a fair bit of fussing with CMake, but I’ve successfully compiled Caffe from the Berkeley Vision and Learning Center. There does appear to still be an issue with a psudorandom generator in one of the linked libraries that will probably impair training new nets, but I can currently run existing models and compare images against them for classification.

I’ll outline some of the issues I ran into along the way.

First of all, always check the installation requirements before starting. Per the installation guide:
CUDA 7+ is required for GPU mode
BLAS (Basic Linear Algebra System) – OpenBLAS libraries are downloaded with the dependency install, don’t need to install separate
BOOST – this is a C++ cross compilation libary, don’t install – python dependency script will get it
protobuf, glog, gflags, hdf5 – Don’t install any of these either!
OpenCV – Script will fetch this too
leveldb, lmdb – Script gets one of these

cuDNN – This one you do need to download for an nVidea GPU setup. Ensures fastest operation

Why are these listed in the requirements if you don’t need to install them!? Well if you were compiling from scratch in linux you’d need to get all these installed first.

Python is also listed as optional, but you’ll need it to run the dependency install scripts. Get Anaconda 2.7. 3.5 doesn’t appear to be supported at the moment. Also download CMake 3.4+. I used 3.6.

OK, got CUDA 8, cuDNN 5, and Anaconda 2.7, and CMake 3.4+ installed? Now start following these windows install instructions.

First issue I ran into was a conflicting python install trying to install dependencies since I have probably at least 5 python installs on my machine.
C:\Programming\caffe>py -2.7
File "C:\tools\python\lib\site.py", line 176
file=sys.stderr)
^
SyntaxError: invalid syntax

Double check your %PATH%, user PATH, PYTHONPATH, PYTHONHOME variables. Why do they have so many variables? After scrubbing that I used the Visual Studio 2015 dependency install from the Caffe root.
> python scripts\download_prebuilt_dependencies.py --msvc_version=v140

I initially tried using the Ninja generator since it supposedly will build faster. Eventually gave up. One note on that process was that the command prompt needs to run in Admin Mode to use conda install commands at least on Windows8.1

I used the “Visual Studio 14 2015 Win64” CMAKE_GENERATOR. The CMAKE generation command I eventually got to work was
cmake -G%CMAKE_GENERATOR% -DBLAS=Open -DCMAKE_BUILD_TYPE=%CMAKE_CONFIGURATION% -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX="C:/Programming/caffe" -DCUDNN_ROOT="D:/Programming/cudnn-8.0-windows7-x64-v5.1/cuda" -DBoost_COMPILER=-vc140 -DBoost_DEBUG=ON -DBoost_USE_STATIC_LIBS=OFF -DBUILD_python_layer=1 -DBUILD_python=1 -DCUDA_gpu_detect_output=FALSE -DCPU_ONLY=0 -DCUDA_VERBOSE_BUILD=ON -DUSE_OPENCV=ON -C %CAFFE_DEPENDENCIES%\caffe-builder-config.cmake ..\

Had a problem where HDF5 library wasn’t found.

CMake Error at C:/Program Files/Anaconda2/Library/cmake/FindHDF5.cmake:85 (include)
include could not find load file:

HDF5_ROOT_DIR-NOTFOUND/hdf5-config.cmake
Call Stack (most recent call first):
cmake/Dependencies.cmake:39 (find_package)
CMakeLists.txt:68 (include)
Delete everything in your CMAKE build directory and try rebuilding. Fixed it for me anyway.

Once I finally got the exe’s built, I also had to move a bunch of DLL’s to the binary directory that Caffe lives in. Not sure if that’s expected or if that’s an incorrect build flag, but Caffe runs after moving them in.

To setup the pycaffe directory you also need to copy the %CAFFE_ROOT%\python\caffe into your site_packages.

After all that I can use the caffe package in IPython and am able to load different caffe models from the community model zoo .

This guide on using pycaffe is very helpful.

Here’s some example out from the InceptionBN-21K-for-Caffe model.
sandwich

In [128]: im = caffe.io.load_image(wd+'examples/images/sandwich.jpg')

In [129]: net.blobs[‘data’].data[…] = transformer.preprocess(‘data’, im)

In [130]: out = net.forward()

In [131]: top_k = net.blobs[‘softmax’].data[0].flatten().argsort()[-1:-6:-1]

In [132]: print labels[top_k]
[‘n07698401 Bacon-lettuce-tomato sandwich, BLT’
‘n07697825 Bomber, grinder, hero, hero sandwich, hoagie, hoagy, Cuban sandwich,
Italian sandwich, poor boy, sub, submarine, submarine sandwich, torpedo, wedge,
zep’
‘n07696403 Sandwich plate’ ‘n07696625 Ham sandwich’
‘n07696839 Club sandwich, three-decker, triple-decker’]

No bacon or ham in there, but pretty good for top 5 results.