RIT CIS Summer Internship

It's hard to believe it's over. This morning we had our presentations in the auditorium, and everyone did really well. I remembered pretty much everything I wanted to say, and I think I explained my topic thoroughly. The presentations ended at almost exactly 11, and after closing remarks, photographs, and conversations, I stayed behind at RIT for a while and sent my files with the final code to my advisors. At noon my lab (which also includes two other graduate students) went out to lunch to celebrate, which was really nice because we ended up talking a lot (and not about debugging!) and I got to know them better. It's a little bittersweet because now my involvement is over, but it was a really rewarding experience and I learned how to do something new (many things actually). Overall, although it was stressful at times, I'm glad I ended up having so much autonomy with my project because it really taught me a lot about myself and about making mistakes. I'm going

The paired setting of the model I tested yesterday was not good at all. I decided not to test any more and instead focused on my presentation. I fixed an image that I realized didn’t match, got some more questions answered, and reread some of the papers from earlier. I also went over each of my slides and practiced presenting.  In the afternoon I discovered that using PSNR for the unpaired settings of the models doesn’t make a lot of sense (I had suspicions earlier but got them confirmed today)- it doesn’t affect the image results, but the graphs I created for those settings are now misleading/incorrect. Oh well, something to add to the reflection. Overall today wasn’t too eventful since it was all preparation. Hopefully I’m ready for tomorrow!

I tested the model from yesterday and it looks surprisingly similar to the best one- the color of the sky is a little split between pink and blue but is blended in most of the images, and overall the objects look a little sharper. The graphs for the loss don’t make that much sense though, and the PSNR for the RGB images fluctuates a lot. This morning we ran through our presentations in the auditorium. We started at 9 am and didn’t finish until 11:50, but we had a lot of time in between each one for feedback. Mine was second and went 16 minutes (better time than I thought) and I got a lot of good advice too. Sometimes I forget that other people haven’t seen my presentation/topic before so I have to remember to explain/define everything. At lunch Jocelyn, Hannah, and I went out to eat and then to Abbott's. We also saw a beaver by the RIT entrance!  In the afternoon I worked on my presentation some more- I cut out some of the graphs as Joe suggested and also made the diagrams clear

I tested some more models today- no significant improvements. But I got to meet with both of my advisors in the morning for at least an hour to go over my presentation so I got a lot of good feedback and also some explanations for why the results haven’t been that great. It turns out that the U-Net structure is ok for RGB-to-thermal translation, which is reflected in the graphs and output images, but not so great for thermal to RGB because that translation ends up with increased complexity/texture. Translating from thermal to RGB accurately is difficult on its own, but with a model and/or loss function not suited to that translation specifically, it could mislead the training. That could also explain why the results using the paired dataset are worse- it’s not making the right connections between the two modalities and when it only has the exact image to compare to rather than something more varied it actually makes it worse. I also got suggestions to improve the image quality like add

We started having morning meetings again. I started the day by testing the three models waiting for me. Two were pretty bad, one was ok. I’m running two more right now, but I’m not sure what I should be doing to make them improve. Right now I'm changing how often the networks are optimized to hopefully balance them more. I spent most of the day working on my presentation by organizing the graphs and images I want to show. They’re not great, but if they’re the best I can do then so be it. In the afternoon I had another meeting where Joe went over David's presentation and then mine to give feedback and see if there was any overlap in background information. I hope I can meet with my advisors soon to go over my presentation as well.

The three models were done training by this morning, and I tested all of them- they were ok, but the paired dataset ones still have splotches on them. Some of the generator losses were actually increasing after a certain number of epochs (around 11-12) so I decided to train another one with the paired dataset for 12 epochs. I’m also retraining the best unpaired one I have so far and also the paired version for the original model because I still don’t have the RGB to thermal images. However I might end up having to retrain them because I might have messed up the saved files again. There are so many of them! I also think that some of the issues might be due to overfitting so I added more dropouts to the model and starting running that too. I can train four right now because I managed to get one of them onto Grissom. Eventually I'm going to change more of the hyperparameters, and I'm also thinking of optimizing the generator more often. I can always run more at home and get the

I had some minor computer issues today so I didn’t get to test that much. Yesterday I put the desktop into suspension (which I now know I’m not supposed to do) and couldn’t get it back on, so I tried using a screen at home and detaching it but then I wasn’t able to reconnect at RIT today (probably because of the Matplotlib graphs). In any case I had to restart the training this morning. Early in the afternoon, I put the computer into lock because that was supposed to be the proper way to do it but it didn’t work and in bringing the computer back to life my training was lost again. Basically it still hasn't finished, but I'll get results tomorrow! Hopefully they look better… testing right now it doesn't look too promising- the ones using the paired dataset are still looking strange and the ones using the unpaired dataset pretty much only get the sky the right color... but it was only after a few epochs so it might still improve. I also met with Joe in the afternoon to get so

When I came in today, two of the models (the second and third one from yesterday) were still training, but the first one had stopped after the 20th epoch due to a “CUDA error: invalid device ordinal.” So I spent the morning working on my presentation and figuring out ways to change the code. The models didn’t finish until 3:30 pm, and it turns out that 50 epochs is not the way to go. Especially for the paired dataset one… not good. I tested the one that got stopped at 20 epochs and the outputs were decent (the color of the sky is closer), so I’m running that one with the unpaired dataset again for 20 epochs and also for 10 epochs. I’m also running it using the paired dataset for 20 epochs (not enough memory to do the 10 epoch one but that should be fast tomorrow). I’m not sure what the issue is with the paired image results, because they should be better than the unpaired, but it seems that the discriminator outperforms the generators early on, or there could be overfitting occurring

When I came in this morning all three of the models were done training. After testing each of them and seeing the results, I had to abandon the simplified model since it seems to think the sky is red (among other problems). (Ahhhhhh)               The original model fared better. Here’s a result from using the unpaired dataset. The colors are pretty dull and there are weird outlines between the sky and the trees/buildings but it looks better at least. And here's a RGB to thermal translation. I realize I should probably include the ground truth for all of these. There are thousands of testing images, but I might cut it down so it’s easier for me to match up images for the presentation. As for using the paired dataset, the outputted images somehow look worse and the graphs from the training were also strange. Most of the images have white dots over them, and while there are warmer colors it lost the blue… I’m wondering if I need to change the model more for it to

Last Friday my advisors showed me how to connect to the RIT servers from home so I worked on my project over the weekend. After deconstructing my training code and reconstructing it again I finally figured out it was the metric calculations that were causing the accumulating memory usage. After fixing it I was able to run the code once on the simplified model (however I didn’t get any graphs from it because of an error), but now that I'm sure it’s working I’m going to train the original model. The system administrator also taught me how to use a screen session so I can detach the computer and still have the code running in the background (in case the computer loses connection, which happened twice at home). Right now I’m training three models at the same time- it’s slower (it’s been almost 7 hours and the one with the simplified model has gone through around 15/30 epochs and the ones with the original model have gone through around 10/30 epochs) but I think it’ll still take less ti

I realized this morning that the problem might not just be with the lack of memory (foreshadowing...) so I spent the morning cutting down the amount of memory my model was using by simplifying the network layers and making the crop, batch size, etc. smaller. But as I found out later, I think the issue is with my training code rather than the model. At 1 pm I went to a master’s thesis defense by one of the students in the visual perception lab. Her presentation was very interesting and also engaged the audience. Later in the afternoon the system administrator finished installing the GPU, but when I tried to run my program it still used up too much memory, which was surprising and a little concerning. The memory usage is supposed to plateau after a certain point, but mine just keeps accumulating until it reaches the maximum. I'm looking into different reasons as to why that's happening, and I've gotten the accumulation to slow (but not stop yet). I was hoping to train ov

I mostly spent the day trying to find a place to train my model. First the system administrators set up a CIS account for me so I could access Grissom, but after many technical issues (computer wouldn’t let me use Grissom, screen froze, etc.) it turns out that Grissom doesn't have enough memory either because of all the people using it (but I appreciate their efforts). Ultimately, we decided the best solution was to install a new GPU into a machine the system administrators have lying around and using that to run my code. They said it should be done by tomorrow afternoon, so hopefully I can train over the weekend.  Today was also the Undergraduate Research Symposium. I went with Jocelyn during lunch and heard the keynote speaker, Jason Babcock, talk about his research and design of eye-trackers, his company, Positive Science, and his experiences at RIT. Later in the afternoon, we also went to look at the posters and listen to an oral presentation (I was intrigued because part of t