tag:blogger.com,1999:blog-6141980.comments2017-11-08T22:46:37.059-06:00Nuit BlancheIgorhttp://www.blogger.com/profile/17474880327699002140noreply@blogger.comBlogger1505125tag:blogger.com,1999:blog-6141980.post-63276383419939607192017-11-03T19:24:51.492-05:002017-11-03T19:24:51.492-05:00Offtopic: Chaos cancelling neural network ensemble...Offtopic: Chaos cancelling neural network ensembles.<br />The idea is that the non-linearities in deep neural networks compound (exponentially) layer after layer. A recent paper shows single pixel attacks of deep networks that would support the idea of bifurcations along 1 or several dimensions. Implying chaos theory applies.<br />By using ensembles of diverse neural networks you should be able to cancel out chaotic responses to low level Gaussian noise according to the central limit theorem.<br /><br />It shouldn't add too much extra computational burden because if you train the ensemble collectively you still get a chaos cancelling effect using individual networks with fewer weight parameters each. <br />https://groups.google.com/forum/#!topic/artificial-general-intelligence/itUghRNZWN8Sean O'Connorhttps://www.blogger.com/profile/16235434542222391913noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-4587269790536423902017-09-30T14:18:19.880-05:002017-09-30T14:18:19.880-05:00Kevin,
The videos of either Mark Davenport or Emm...Kevin,<br /><br />The videos of either Mark Davenport or Emmanuel Candes do a pretty good job I think. <br /><br />Cheers,<br /><br />Igor.Igorhttps://www.blogger.com/profile/17474880327699002140noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-85223273754953120602017-09-30T11:57:54.689-05:002017-09-30T11:57:54.689-05:00Thanks Igor,
I actually did take a look at that ...Thanks Igor, <br /><br />I actually did take a look at that page. Your page here (https://nuit-blanche.blogspot.com/p/teaching-compressed-sensing.html) is an excellent resource for approaching this topic, providing further reading for people of all levels of mathematical sophistication. Will poke around on that page to see if there is something that I can sink my teeth into. Although I remember a bit of linear algebra, probability, and signal processing from my electrical engineering college days, that was a long time ago (Rice '78).<br /><br />Best,<br />KevinKevin Finchhttps://www.blogger.com/profile/12292237902904082859noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-65992644295006482062017-09-29T18:05:10.794-05:002017-09-29T18:05:10.794-05:00Hi Kevin,
Maybe this explanation will help ( http...Hi Kevin,<br /><br />Maybe this explanation will help ( https://www.quora.com/What-is-compressed-sensing-compressive-sampling-in-laymans-terms/answer/Igor-Carron ) . The number of defective items (1 ball) is small (sparse) compared to the total number of balls (12).<br /><br />Hope this helps,<br /><br />Igor. Igorhttps://www.blogger.com/profile/17474880327699002140noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-89712336441039018122017-09-29T16:34:27.637-05:002017-09-29T16:34:27.637-05:00This took me the better part of a day to figure ou...This took me the better part of a day to figure out as well. Was about to give up when I stumbled on the trick. Better late than never. <br /><br />Thanks for the link to the article, since I was wondering the same thing: How does a puzzle like this relate to Compressed Sensing? Given that my BSEE was a long time ago I doubt that I will be able to understand it, but will do my best. (A shoutout to Steve Hsu's Information Processing blog for stimulating my interest in this topic.)<br /><br />A bientôt.Kevin Finchhttps://www.blogger.com/profile/12292237902904082859noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-89308302685217488792017-09-29T00:44:45.508-05:002017-09-29T00:44:45.508-05:00I mentioned over a the Numenta forum that one pers...I mentioned over a the Numenta forum that one perspective on deep neural networks views them as pattern based fuzzy logic.<br />https://discourse.numenta.org/t/artificial-life-concept/2308/11<br />In particular in higher dimension the dot product weighting function used in neural nets acts as a selective filter. Or you can say the dot product weighting produces a low magnitude output for most any random input and only a small number of select input vectors will produce a high magnitude output.<br />https://www.cs.princeton.edu/courses/archive/fall14/cos521/lecnotes/lec11.pdf<br />Sean O'Connorhttps://www.blogger.com/profile/16235434542222391913noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-5061841268433299022017-08-31T19:16:04.190-05:002017-08-31T19:16:04.190-05:00https://github.com/S6Regen/EvoNethttps://github.com/S6Regen/EvoNetSeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-16692102985063282022017-06-29T09:34:06.866-05:002017-06-29T09:34:06.866-05:00Maybe subrandom sampling can help with compressive...Maybe subrandom sampling can help with compressive sensing, perhaps being better than purely random sampling:<br />https://en.wikipedia.org/wiki/Low-discrepancy_sequence<br />I suppose there is a good chance it has already been investigated. <br />SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-35430551569191075822017-06-24T18:08:58.622-05:002017-06-24T18:08:58.622-05:00Not at the moment, it looks like.
Igor.Not at the moment, it looks like.<br /><br />Igor.Igorhttps://www.blogger.com/profile/17474880327699002140noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-51100357864574434072017-06-24T14:08:07.014-05:002017-06-24T14:08:07.014-05:00Are slides available for this talk?Are slides available for this talk?Gokulhttps://www.blogger.com/profile/14250642559129421797noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-23978504194151106062017-06-04T07:52:01.408-05:002017-06-04T07:52:01.408-05:00There has been a lack of discussion about binariza...There has been a lack of discussion about binarization in neural networks. Multiplying those +1/-1 values by weights and summing allows you to store values with a high degree of independence. For a given binary input and target value you get an error. You divide the error by the number of binary values and then you simply correct each of the weights by the reduced error taking account of the binary sign. That gives a full correction to get the correct target output. In higher dimensional space most vectors are orthogonal. For a different binary input the adjustments you made to the weights will not align at all. In fact they will sum to Gaussian noise by the central limit theorem. The value you previously stored for the second binary input will now be contaminated by a slight amount of Gaussian which you can correct for. This will now introduce an even smaller amount of Gaussian noise on the value for the first binary input. Iterating back and forth will get rid of the noise entirely for both binary inputs. <br />This has high use in random projection,reservoir and extreme learning machine computing.SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-73081378628405822532017-05-30T19:15:38.529-05:002017-05-30T19:15:38.529-05:00Tested in code:
http://www.freebasic.net/forum/vie...Tested in code:<br />http://www.freebasic.net/forum/viewtopic.php?f=7&t=25710<br />Conclusion: Very good<br /><br />The idea is very applicable to locality sensitive hashing as well.SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-74043354810802289852017-05-30T01:29:57.391-05:002017-05-30T01:29:57.391-05:00It would seem you could fit about 100 million inte...It would seem you could fit about 100 million integer add/subtract logic units on a current semiconductor die. Clock them at 1 billion operations per second and you have 100 Peta operations per second available for "no multiply" nets. <br />https://discourse.numenta.org/t/no-multiply-neural-networks/2361<br />SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-38381881368275629822017-05-30T01:25:34.630-05:002017-05-30T01:25:34.630-05:001 billion by 100 million operations is 100 Peta op...1 billion by 100 million operations is 100 Peta operations per second dude.SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-52153707086744812532017-05-30T00:10:03.992-05:002017-05-30T00:10:03.992-05:00This comment has been removed by the author.SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-13018719162268526322017-05-29T19:05:43.929-05:002017-05-29T19:05:43.929-05:00It should be possible to do "no multiply"...It should be possible to do "no multiply" neural nets using random sign flipping + WHT random projections and the signof function using the architecture in this paper:<br />https://arxiv.org/pdf/1705.07441.pdf<br /><br />In hardware all you would need are low transistor count, low power requirement integer add and subtract operations and a few other simple bit operators. Avoiding much more complex and space consuming multiply logic circuit. It should be quite easy to pipe-line the operations on a FPGA. One other thing is that you may not need full precision integer +- because 2's complement overflow would simply increase the amount of nonlinearity, but probably not too much. SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-85552078723309688112017-05-23T17:16:45.628-05:002017-05-23T17:16:45.628-05:00I noticed this paper entitled "Exponential Ca...I noticed this paper entitled "Exponential Capacity in an Autoencoder Neural Network with a Hidden Layer."<br />https://arxiv.org/pdf/1705.07441.pdf<br />I would kinda guess the exponential capacity is because real value weights are used to encode the binary output. With finite precision arithmetic say, 16 bit half floats or 32 bit floats there probably is an optimal number of weights to sum together to get a result. After that you should likely use locality sensitive hashing to switch in different weight vectors. I also noticed recently that the new nVidia GPU chip offers a 120 Tflop half float matrix operation that might be suitable for random projections. If you were lucky you might get say 40 million 65536-point RPs per second out of it. That would be about 4000 times faster than I can get from my dual core CPU using SIMD. SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-11507414279483606542017-05-19T12:52:24.840-05:002017-05-19T12:52:24.840-05:00Does it beat HYPERBAND though?Does it beat HYPERBAND though?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6141980.post-59510053660011431762017-05-16T04:17:42.032-05:002017-05-16T04:17:42.032-05:00I'll read that paper later:
On this video abou...I'll read that paper later:<br />On this video about lasso:<br />https://youtu.be/Hn8NtydkeDs<br /><br />I made this comment:<br /><br />"You are saying that the reconstructed data lies on an L1 manifold. You can learn a manifold using say a single layer neural network autoencoder. Then to reconstruct you can invert the dimensionally reduced data, get the autoencoder to correct it, send it back through the dimensional reduction and correct only the reduced aspect. Just bounce back and forth between the two.<br />Or you could set the manifold to be the moving average of the data which is a very easy manifold to correct to and bounce between the two. Anyway: https://drive.google.com/open?id=0BwsgMLjV0BnhOGNxOTVITHY1U28"SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-54112671618408092032017-05-03T23:33:15.994-05:002017-05-03T23:33:15.994-05:00There is also an algorithm tsunami, not just a dat...There is also an algorithm tsunami, not just a data one!!!<br />Another possibility would be to do computational self-assembly of neural nets. <br />http://www.exa.unicen.edu.ar/escuelapav/cursos/bio/l21.pdf SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-10224157172252653502017-05-03T11:05:15.590-05:002017-05-03T11:05:15.590-05:00Interesting bunch of articles Igor!
Cheers,
RaviInteresting bunch of articles Igor!<br /><br />Cheers,<br />RaviRavi Kiranhttps://www.blogger.com/profile/02116578557275934638noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-53489892968226296242017-05-02T17:30:36.675-05:002017-05-02T17:30:36.675-05:00It would be interesting to see this with Resnets t...It would be interesting to see this with Resnets too.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6141980.post-31426580021222435732017-04-26T00:38:09.441-05:002017-04-26T00:38:09.441-05:00I think what is happening is that you are getting ...I think what is happening is that you are getting unsupervised feature learning in the deeper layers and then one final readout layer. That may give a boost in performance in some circumstances. There probably are better ways to do unsupervised feature learning prior to a readout layer. There are also some aspects to do with noise and maybe some cooling effect over time as the system adapts. Thumbs up or thumbs down, I don't know. You be the judge. <br /> <br />SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-83470436554465103092017-04-25T23:45:22.461-05:002017-04-25T23:45:22.461-05:00Re: https://openreview.net/pdf?id=HkXKUTVFl
I'...Re: https://openreview.net/pdf?id=HkXKUTVFl<br />I'm trying dropout in relation to the back error projection. Anyway there are tons of ideas to explore, especially if you start using fast random projection algorithms for both the back error projection and the forward aspects of a network.SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-81323322034417661492017-04-24T07:51:28.809-05:002017-04-24T07:51:28.809-05:00Neat, will try.Neat, will try.SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.com