tag:blogger.com,1999:blog-6141980.comments2018-04-18T15:17:09.188-05:00Nuit BlancheIgorhttp://www.blogger.com/profile/17474880327699002140noreply@blogger.comBlogger1514125tag:blogger.com,1999:blog-6141980.post-57481468020649726342018-04-18T15:15:38.155-05:002018-04-18T15:15:38.155-05:00Thanks, Nuit!Thanks, Nuit!Cun Muhttps://www.blogger.com/profile/04865641672431083968noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-80789143665042749382018-02-14T04:34:38.450-06:002018-02-14T04:34:38.450-06:00toutes les infos pour s'inscrire sont ici: htt...toutes les infos pour s'inscrire sont ici: https://www.meetup.com/Paris-Machine-learning-applications-group/events/241149416/Igorhttps://www.blogger.com/profile/17474880327699002140noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-44175179632755434212018-02-14T04:30:54.047-06:002018-02-14T04:30:54.047-06:00Ca aura lieu où cet événement?Ca aura lieu où cet événement?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6141980.post-87135115121772221652018-02-07T16:03:40.834-06:002018-02-07T16:03:40.834-06:00It seems that was an early/incomplete draft which ...It seems that was an early/incomplete draft which was also updated quickly after several days.<br />It is now much more through than it was initially, suggesting either the guy didn't know how Arxiv works in first place or somehow it got published before the he (anyone responsible for publishing it) realizes it!<br />Anyway, the latest version looks promising IMHO though. Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6141980.post-6782081726706041972018-01-30T17:43:20.758-06:002018-01-30T17:43:20.758-06:00One other thing I should mention is information lo...One other thing I should mention is information loss by non-linear functions. Linear transformation can always be organized to leave the information content almost completely intact. With standard floating point operations (IEEE) any non-linear (activation) function will always cause information loss (ie the first derivative will be very coarse):<br /><br />https://www.spsc.tugraz.at/system/files/Geiger_InfoLossStatic.pdf<br /><br />Layer after layer the non-linearity in a deep neural network compounds. And at each layer a significant amount of information about the input is lost. After a number of layers the network ends up on a set trajectory, the input information is no longer available (has been washed out) to cause further switching between decision regions. <br /><br />In the system discribed maybe the random pathways through the network allow information about the input to pass through less hindered than if all the weights had been organized.<br /><br />SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-36166645609672676372018-01-29T17:43:53.271-06:002018-01-29T17:43:53.271-06:00I think the idea with generalization is to make th...I think the idea with generalization is to make the decision regions as large as possible consistent with the training data. Tishby noted a diffusion process occurring in neural networks trained by backpropagation leading to compression. <br />https://youtu.be/bLqJHjXihK8<br />I presume that diffusion process increases the size of the decision regions maximally.<br /><br />I suppose just having a lot of random weights around automatically causes some broadening of the decision regions. Anyway it is interesting information for me. It provides some justification for evolving subsets of weights in a neural network rather than trying to evolve them all at once. <br />There is an interesting question whether or under what circumstances evolution can provide a similar diffusion process to backpropagation. <br /> SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-52504343890977950092018-01-12T00:32:40.660-06:002018-01-12T00:32:40.660-06:00Off topic, somewhat:
https://github.com/S6Regen/Da...Off topic, somewhat:<br />https://github.com/S6Regen/Data-Reservoir-AI<br />It does make heavy use of random projections. SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-56968162608666517542017-12-18T14:43:14.426-06:002017-12-18T14:43:14.426-06:0022/12 : first day of vacations. I hesitate between...22/12 : first day of vacations. I hesitate between crowded malls and this workshop... No moreLaurent Duvalhttps://www.blogger.com/profile/05343286920474971263noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-30865629846668953292017-12-17T22:07:11.990-06:002017-12-17T22:07:11.990-06:00Will this be streamed or a video taken for later d...Will this be streamed or a video taken for later dissemination?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6141980.post-63276383419939607192017-11-03T19:24:51.492-05:002017-11-03T19:24:51.492-05:00Offtopic: Chaos cancelling neural network ensemble...Offtopic: Chaos cancelling neural network ensembles.<br />The idea is that the non-linearities in deep neural networks compound (exponentially) layer after layer. A recent paper shows single pixel attacks of deep networks that would support the idea of bifurcations along 1 or several dimensions. Implying chaos theory applies.<br />By using ensembles of diverse neural networks you should be able to cancel out chaotic responses to low level Gaussian noise according to the central limit theorem.<br /><br />It shouldn't add too much extra computational burden because if you train the ensemble collectively you still get a chaos cancelling effect using individual networks with fewer weight parameters each. <br />https://groups.google.com/forum/#!topic/artificial-general-intelligence/itUghRNZWN8Sean O'Connorhttps://www.blogger.com/profile/16235434542222391913noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-4587269790536423902017-09-30T14:18:19.880-05:002017-09-30T14:18:19.880-05:00Kevin,
The videos of either Mark Davenport or Emm...Kevin,<br /><br />The videos of either Mark Davenport or Emmanuel Candes do a pretty good job I think. <br /><br />Cheers,<br /><br />Igor.Igorhttps://www.blogger.com/profile/17474880327699002140noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-85223273754953120602017-09-30T11:57:54.689-05:002017-09-30T11:57:54.689-05:00Thanks Igor,
I actually did take a look at that ...Thanks Igor, <br /><br />I actually did take a look at that page. Your page here (https://nuit-blanche.blogspot.com/p/teaching-compressed-sensing.html) is an excellent resource for approaching this topic, providing further reading for people of all levels of mathematical sophistication. Will poke around on that page to see if there is something that I can sink my teeth into. Although I remember a bit of linear algebra, probability, and signal processing from my electrical engineering college days, that was a long time ago (Rice '78).<br /><br />Best,<br />KevinKevin Finchhttps://www.blogger.com/profile/12292237902904082859noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-65992644295006482062017-09-29T18:05:10.794-05:002017-09-29T18:05:10.794-05:00Hi Kevin,
Maybe this explanation will help ( http...Hi Kevin,<br /><br />Maybe this explanation will help ( https://www.quora.com/What-is-compressed-sensing-compressive-sampling-in-laymans-terms/answer/Igor-Carron ) . The number of defective items (1 ball) is small (sparse) compared to the total number of balls (12).<br /><br />Hope this helps,<br /><br />Igor. Igorhttps://www.blogger.com/profile/17474880327699002140noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-89712336441039018122017-09-29T16:34:27.637-05:002017-09-29T16:34:27.637-05:00This took me the better part of a day to figure ou...This took me the better part of a day to figure out as well. Was about to give up when I stumbled on the trick. Better late than never. <br /><br />Thanks for the link to the article, since I was wondering the same thing: How does a puzzle like this relate to Compressed Sensing? Given that my BSEE was a long time ago I doubt that I will be able to understand it, but will do my best. (A shoutout to Steve Hsu's Information Processing blog for stimulating my interest in this topic.)<br /><br />A bientôt.Kevin Finchhttps://www.blogger.com/profile/12292237902904082859noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-89308302685217488792017-09-29T00:44:45.508-05:002017-09-29T00:44:45.508-05:00I mentioned over a the Numenta forum that one pers...I mentioned over a the Numenta forum that one perspective on deep neural networks views them as pattern based fuzzy logic.<br />https://discourse.numenta.org/t/artificial-life-concept/2308/11<br />In particular in higher dimension the dot product weighting function used in neural nets acts as a selective filter. Or you can say the dot product weighting produces a low magnitude output for most any random input and only a small number of select input vectors will produce a high magnitude output.<br />https://www.cs.princeton.edu/courses/archive/fall14/cos521/lecnotes/lec11.pdf<br />Sean O'Connorhttps://www.blogger.com/profile/16235434542222391913noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-5061841268433299022017-08-31T19:16:04.190-05:002017-08-31T19:16:04.190-05:00https://github.com/S6Regen/EvoNethttps://github.com/S6Regen/EvoNetSeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-16692102985063282022017-06-29T09:34:06.866-05:002017-06-29T09:34:06.866-05:00Maybe subrandom sampling can help with compressive...Maybe subrandom sampling can help with compressive sensing, perhaps being better than purely random sampling:<br />https://en.wikipedia.org/wiki/Low-discrepancy_sequence<br />I suppose there is a good chance it has already been investigated. <br />SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-35430551569191075822017-06-24T18:08:58.622-05:002017-06-24T18:08:58.622-05:00Not at the moment, it looks like.
Igor.Not at the moment, it looks like.<br /><br />Igor.Igorhttps://www.blogger.com/profile/17474880327699002140noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-51100357864574434072017-06-24T14:08:07.014-05:002017-06-24T14:08:07.014-05:00Are slides available for this talk?Are slides available for this talk?Gokulhttps://www.blogger.com/profile/14250642559129421797noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-23978504194151106062017-06-04T07:52:01.408-05:002017-06-04T07:52:01.408-05:00There has been a lack of discussion about binariza...There has been a lack of discussion about binarization in neural networks. Multiplying those +1/-1 values by weights and summing allows you to store values with a high degree of independence. For a given binary input and target value you get an error. You divide the error by the number of binary values and then you simply correct each of the weights by the reduced error taking account of the binary sign. That gives a full correction to get the correct target output. In higher dimensional space most vectors are orthogonal. For a different binary input the adjustments you made to the weights will not align at all. In fact they will sum to Gaussian noise by the central limit theorem. The value you previously stored for the second binary input will now be contaminated by a slight amount of Gaussian which you can correct for. This will now introduce an even smaller amount of Gaussian noise on the value for the first binary input. Iterating back and forth will get rid of the noise entirely for both binary inputs. <br />This has high use in random projection,reservoir and extreme learning machine computing.SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-73081378628405822532017-05-30T19:15:38.529-05:002017-05-30T19:15:38.529-05:00Tested in code:
http://www.freebasic.net/forum/vie...Tested in code:<br />http://www.freebasic.net/forum/viewtopic.php?f=7&t=25710<br />Conclusion: Very good<br /><br />The idea is very applicable to locality sensitive hashing as well.SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-74043354810802289852017-05-30T01:29:57.391-05:002017-05-30T01:29:57.391-05:00It would seem you could fit about 100 million inte...It would seem you could fit about 100 million integer add/subtract logic units on a current semiconductor die. Clock them at 1 billion operations per second and you have 100 Peta operations per second available for "no multiply" nets. <br />https://discourse.numenta.org/t/no-multiply-neural-networks/2361<br />SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-38381881368275629822017-05-30T01:25:34.630-05:002017-05-30T01:25:34.630-05:001 billion by 100 million operations is 100 Peta op...1 billion by 100 million operations is 100 Peta operations per second dude.SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-52153707086744812532017-05-30T00:10:03.992-05:002017-05-30T00:10:03.992-05:00This comment has been removed by the author.SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.comtag:blogger.com,1999:blog-6141980.post-13018719162268526322017-05-29T19:05:43.929-05:002017-05-29T19:05:43.929-05:00It should be possible to do "no multiply"...It should be possible to do "no multiply" neural nets using random sign flipping + WHT random projections and the signof function using the architecture in this paper:<br />https://arxiv.org/pdf/1705.07441.pdf<br /><br />In hardware all you would need are low transistor count, low power requirement integer add and subtract operations and a few other simple bit operators. Avoiding much more complex and space consuming multiply logic circuit. It should be quite easy to pipe-line the operations on a FPGA. One other thing is that you may not need full precision integer +- because 2's complement overflow would simply increase the amount of nonlinearity, but probably not too much. SeanVNhttps://www.blogger.com/profile/05967727000105480078noreply@blogger.com