IRS Scandal - Lost Emails

[Logan Darklighter]

Well color me unsurprised. Nothing this criminal government does surprises me any more, except the depths to which they seem willing to descend. There's no "lowest level" to any of this. 

No Wonder the IRS Is ‘Losing’ E-mails: It Was Trying to Throw Innocent Conservatives in JailBy David FrenchEven a small child can connect these dots.
The IRS is announcing the “loss” of mass numbers of e-mails (do they have any computers that don’t crash?) even as the e-mails that do exist are beginning to show the extent of IRS corruption. Let’s take this exchange (previously uncovered byJudicial Watch) between Lois Lerner, the director of exempt organizations at the IRS, and Nikole Flax, then the IRS commissioner’s chief of staff. (To be clear, these are not “low level” employees.)
First, here’s Lerner on May 8, 2013, literally two days before last year’s fake apology for IRS tea-party targeting:

I got a call today from Richard Pilger Director Elections Crimes Branch at DOJ. I know him from contacts from my days there. He wanted to know who at IRS the DOJ folk s could talk to about [Rhode Island Democrat] Sen. Whitehouse idea at the hearing that DOJ could piece together false statement cases about applicants who “lied” on their 1024s –saying they weren’t planning on doing political activity, and then turning around and making large vis iblepolitical expenditures. DOJ is feeling like it needs to respond, but want to talk to the right folks at IRS to see whether there are impediments from our side and what, if any damage this might do to IRS programs.
I told him that sounded like we might need several folks from IRS. I am out of town all next week, so wanted to reach out and see who you think would be right for such a meeting and also hand this off to Nan as contact person if things need to happen while I am gone –

Translation: The Obama Justice Department was reaching out to the Obama IRS to see if it could “piece together” prosecutions of nonprofits even before any evidence of wrongdoing emerged.
And how did Nikole Flax respond? By suggesting that even more federal agencies get involved:

I think we should do it – also need to include CI, which we can help coordinate. Also, we need to reach out to FEC. Does it make sense to consider including them in this or keep it separate?

In other words, rather than asking the DOJ whether it possessed any evidence of wrongdoing by American citizens engaged in constitutionally protected activities, two senior officials in the Obama IRS (including one, Lois Lerner, who “joked” about wanting to work for Organizing For Action, also known as BarackObama.com) pushed forward the effort to launch a multi-agency criminal probe, involving even the FEC. Oh, and keep in mind that this e-mail exchange occurred literally years after the IRS claimed it had ended all tea-party targeting.
And in the last few days we learn the IRS has “lost” e-mails from — among others — Lois Lerner and Nikole Flax. What a remarkable coincidence.
The scale of the wrongdoing is staggering.

[robkelk]

And?

This pales beside the "lost government emails" matter where I live - it doesn't cost anyone even one billion dollars, let alone what out mess cost.
--
Rob Kelk
"Governments have no right to question the loyalty of those who oppose
them. Adversaries remain citizens of the same state, common subjects of
the same sovereign, servants of the same law."

- Michael Ignatieff, addressing Stanford University in 2012

[Labster]

Reading the quoted text of the emails, I'm feeling outraged that the IRS didn't initiate prosecutions.  If groups lied on the the intent or extent of political activities, that's a crime.  Period (being the only "dot" I can see).
I think the IRS is clearly in the wrong on the whole affair.  Section 501(c)4 of the law clearly states that those organizations should be operated exclusively for charitable purposes.  The IRS redefined that with a rule that says that they can be operated primarily for charitable purposes.  Law trumps rules, and 50% != 100%.  Clearly, all of these political 501(c)4 organizations are in fact illegal, liberal and conservative alike.
In other news,http://arstechnica.com/tech-policy/2014 ... s-e-mails/10/10 excellent trolling, sir.
-- ∇×V

[Logan Darklighter]

James Poulos isn't the first to draw notice to the historical resonance of an unexplained gap in records in the middle of a probe about abuse of power in the executive branch. Michael Ramirez also made the same point yesterday in his editorial cartoon for Investors Business Daily:

Poulos provides a cautionary voice to both Republicans and Democrats in this instance. Just because the e-mails are missing doesn't mean that malice and corruption have been proven— but it certainly looks that way, and Democrats take a big risk in continuing to call this a “phony scandal” now:

Reminiscent of nothing so much as Nixon secretary Rose Mary Woods’s infamous selective erasures of her boss’s tapes, today’s sought-after communiqués are, we are told, victim of a crashed computer and a failed hard drive, both belonging to Lois Lerner, the chief of the IRS tax-exempt office at the center of the controversy.
And in good Nixonian style, it’s not just Lerner whose emails have gone unaccounted for. Six more IRS employees connected to the original targeting outrage have records the agency cannot submit to Congress—including Nikole Flax, chief of staff to the man eventually fired for his role in targeting conservative groups while serving as acting commissioner.
Best of all, from a salacious standpoint, congressional investigators now know that the IRS kept the missing emails a secret since perhaps February of this year. Lies! Omissions! A cover-up! The possibilities are endless, and weirdly reassuring in these disjointed and murky times. After such a long drumbeat of bad news seemingly “from the future”—from drones and NSA surveillance to fresh chaos in Iraq—the classic, petty terrain of the IRS scandal returns us to comfortably familiar ground. …
Fortunately for the GOP, Americans have an instinctive appreciation for the seriousness of the charges facing the IRS. Putative scandals like the Benghazi incident strike many of us as not just farfetched but almost, as they put it inSpinal Tap, “best left unsolved.” The realm of foreign policy is a realm of mystery for most Americans, a place where government schemes will never be fully revealed and the alternative to trust is a fruitless journey into rabbit-hole speculation.
The possibility of being abused by the IRS, however, requires no such paranoia. Perhaps few of us believe the IRS is really after us in particular; the more general idea that the IRS treats us unequally, in part for political reasons, strikes at the core of reasonable, mainstream distrust of government. After all, the power to tax is the power to destroy. If there’s one place we’d expect to find government harassment carried out under our noses, it’d be the IRS, which harasses us year in and year out anyway.

The foreign-policy crises and failure may not resonate with many Americans who feel distanced from those experiences. That cannot be said for the IRS, the ubiquitous presence that worries all Americans and has a reputation for harsh enforcement along the guilty-until-proven-innocent paradigm. In Ed Morrissey's column today for The Week, he argues that the disappearance of records from an agency that would hardly tolerate such excuses from any other American is the kind of smoking hypocrisy that will keep this scandal alive:

This gives the scandal new and legitimate legs, for a couple of reasons. First, despite having demanded these records from the IRS for over a year, the agency waited until now (and in a Friday afternoon document dump, no less) to inform Congress of the supposed loss of emails. That makes it look very suspicious, and put together with Lerner’s refusal to testify, even more so.
The second reason is that the IRS is the one agency that demands everyone else keep spotless records for seven years or more on their returns. Now we find out that they’re only keeping their own documentation for six months? For a nation founded on the rule of law and equality under it, this retention for thee but no for we will likely offend a lot more people than extra scrutiny for conservative tax-exempt applicants did, and the lame dog ate my homework excuse will offend the rest.
Sadly, the rule of law seems to be the biggest fantasy of all in the case of the IRS targeting scandal and the abuse of power it represents.

Also - anybody who knows anything at all about how emails are sent, stored, transmitted and received knows that the "emails are missing" line is ridiculous. They exist on multiple servers. You cannot just "lose" them. That's not the way the system works or has ever worked. So this "excuse" rings amateurishly false to almost everyone, no matter what end of the political spectrum you may be - liberal or conservative. 

[Rajvik]

Thank you for that last aside Logan, I wasn't quite sure how emails were stored. I always assumed a central server (with backups) handled it for each email company.
 

[Rev Dark]

As someone who knows quite a bit about how e-mails are sent and stored; Logan's description is not entirely accurate.
Depending on the security of the information involved and the architecture of the system, the e-mails may have existed on a single server, or the mailbox on a single server.  Usually in an enterprise implementation with thousands of addresses a fixed number of addresses will be assigned to each physical (or virtual device).  That device may or may not be mirrored or hot-sited for redundancy.   This means that a SPOF incident can wipe out an entire mailbox, or series of mailboxes.  That leaves backups.  The great millstone about the neck of any organization.  Backups are only as good as the last confirmed physical restore of the data contained therein.  For mail servers, depending on the implementation, this gets even scarrier, as restoration of a previous version of a mailbox often requires recreation of the entire server and all contents -rather than that one mailbox.  For this reasons, the restores are not often tested as often as they should be (cough! almost never! cough!)
So this may be deliberate, it may be that with years of budget cuts, the IRS has not been able to invest fully in the maintenance and renewal of their mail architecture.
These are details that will come out in time.

[Bob Schroeck]

Oh, no, net mail is http://wordtothewise.com/2009/03/email ... d-forward/]store-and-forward -- every server between sender and recipient should have a copy, at least for a little while. It's only between "adjacent" systems that the email is directly handed off from origin to destination. This is obscured (by design) in email as it exists today -- the mail system handles the routing transparently -- but in the primordial days of UUCP you used to have to specify the entire route yourself as the address for your mail -- the old http://www.catb.org/jargon/html/B/bang-path.html]"Bang Path" system, so called because you basically listed every machine between you and the recipient, separated by "!"s (like "MyDepartmentServer!MainServer!BigRouter!TheirDepartmentServer!SpecificServer!jsmith").

So yeah, the IRS claiming the emails were lost is pretty disingenuous. About the only way the emails could really be completely gone is if someone went in and deleted them from every system they ever resided on. And if they did that, they might as well be wearing signs around their necks saying "We've got something to hide".
-- Bob
---------
Then the horns kicked in...
...and my shoes began to squeak.

[Logan Darklighter]

The major points being - 1 - It's highly likely (almost certain, in fact) that the government uses some variant of Microsoft Exchange for their email servers. They have built-in exchange mail database redundancy. So, unless they did not follow Microsoft's recommendations they are lying. Check out the diagram below for a basic example. If you have three servers in a DAG you have three copies of the database.



2 - Every IT organization that I know of has hotswappable disk drives. Every server built since 2000 has them. Meaning that if a single disk goes bad it’s easy to replace.
3 - ALL Servers use some form of RAID technology. The only way that data can be totally lost (Meaning difficult to bring back) is if more than a single disk goes before the first bad disk is replaced. In the diagram below you can see that its possible to lose a single disk and still keep the data.


4 - If the server crashed (Hardware failure other than disks), then the disks that contain the DATA for the Exchange database is still available because the server hardware and disks are exchangeable. Meaning that if I have another server with the same hardware in it, I can put the disks in and everything should boot right up.
5 - All email servers in a professional organization use TAPE backup. Meaning if all the above fails, you can restore the server using the TAPE backups.
6 - If they are talking about her local PC, then it’s a simple matter of going to the servers which have the email and getting them from the servers. If the servers have removed the data you can still get them by using the backups of the servers to recover the emails.

I don’t know of any email administrator that doesn’t have at least three ways of getting that mail back. It’s either on the disks or it’s on a TAPE backup someplace or in an archive server. There are at least three ways the government can get those emails.

So - 

They. Are. LYING. 
The only way that they could NOT be lying is if the IRS is totally mismanaged and has the worst IT department ever. 
And we all know that is not only not the case, but they would NEVER accept that kind of excuse from you or anyone other entity (individual or corporate) they have ever audited. 

[khagler]

Logan Darklighter wrote:

---

The only way that they could NOT be lying is if the IRS is totally mismanaged and has the worst IT department ever. 
And we all know that is not only not the case, 

We do? Keep in mind that this is the US Federal Government, the same organization that  brought us healthcare.gov.

[Logan Darklighter]

khagler wrote:

---

Logan Darklighter wrote:

---

The only way that they could NOT be lying is if the IRS is totally mismanaged and has the worst IT department ever. 
And we all know that is not only not the case, 

We do? Keep in mind that this is the US Federal Government, the same organization that  brought us healthcare.gov.

Ah. Point. 
Edit: However - the IRS has in the past made it a point to be thorough about records. They're infamous to a lot of people in this country for being - if not instantly, then inevitably - so meticulous about financial records that it's difficult to hide money from them. If there's any kind of discrepancy in an audit, they're known for pouncing on it. 
Whether that's actually true or not is beside the point in regards to the impression we as a nation have had of this organization up till now. It is at least an image they themselves have encouraged. Well that image has now been blown out of the water. And it's a self-inflicted wound.

[Logan Darklighter]

Another point to consider - One of the articles of impeachment drawn up against President Richard Nixon dealt with his use of the IRS to harass his political enemies - thus precedent has already been established that this is an impeachable offense. If it were proven that Obama ordered the IRS to harass conservative organizations... 

[robkelk]

The major points being -
1 - It's highly likely (almost certain, in fact) that the government uses some variant of Microsoft Exchange for their email servers. They have built-in exchange mail database redundancy. So, unless they did not follow Microsoft's recommendations they are lying.

Following that recommendation costs money - taxpayers' money, in this case.

2 - Every IT organization that I know of has hotswappable disk drives. Every server built since 2000 has them. Meaning that if a single disk goes bad it’s easy to replace.

3 - ALL Servers use some form of RAID technology. The only way that data can be totally lost (Meaning difficult to bring back) is if more than a single disk goes before the first bad disk is replaced. In the diagram below you can see that its possible to lose a single disk and still keep the data.

Assuming the data is still there - RAID is not an archive mechanism.

4 - If the server crashed (Hardware failure other than disks), then the disks that contain the DATA for the Exchange database is still available because the server hardware and disks are exchangeable. Meaning that if I have another server with the same hardware in it, I can put the disks in and everything should boot right up.

Again, assuming the data is still there.

Exchange can be set up to automatically delete all emails older than 90 days (or any other number, but that's a typical setting). If nobody archived them, the emails could be gone without any human intervention at all.

5 - All email servers in a professional organization use TAPE backup. Meaning if all the above fails, you can restore the server using the TAPE backups.

Tape backups are optional under CommVault or TSM, the two largest enterprise-backup solutions actually deployed.

Implementing tape backups costs money - taxpayers' money, in this case.

6 - If they are talking about her local PC, ...

Why in the world would enterprise email be stored on a local PC?

All of this can be explained by simple decisions made to save taxpayers' money. Unless you can prove the contrary, you're in tinfoil-hat territory here.
--
Rob Kelk
"Governments have no right to question the loyalty of those who oppose
them. Adversaries remain citizens of the same state, common subjects of
the same sovereign, servants of the same law."

- Michael Ignatieff, addressing Stanford University in 2012

[Dartz]

RAID is not an archive mechanism.

Especially when a disk in the array fails in such a way that it stays 'online', but still screams out corrupted data. For an entire weekend while everyone's at home. I saw that kill a company.

Anyway, I thought ratfucking was a traditional part of US politics. Using the organs of power to legally surpress your opponents by gerrmandering, redistrcting and otherwise making life as hard for them as possible. The difference between 'acceptable politics' and 'impeachable offence' is who's on the receiving end it seems.
________________________________
--m(^0^)m-- Wot, no sig?

[Logan Darklighter]

robkelk wrote:All of this can be explained by simple decisions made to save taxpayers' money. Unless you can prove the contrary, you're in tinfoil-hat territory here.

Since when has the Federal Bureaucracy ever shied away from spending the taxpayers money? The IRS is one of the most powerful bureaucracies in the Federal Government. Their JOB is based on record-keeping in order to bring the government as much money from the taxpayers as possible. To that end, I'm sure they pretty much have effective carte-blanche in terms of funding. And besides - all of the above measures? A drop in the ocean of federal funding, comparatively speaking. And did you forget about the NSA and its archiving of ALL internet traffic? (Talk about "hoist on your own petard.") 

Those emails exist. They are not gone completely, no matter what Lerner or anyone else says.   
If all else fails, then whoever received and sent emails to Lois Lerner (and others) outside of the IRS - such as the White House - can have their emails subpoenaed. And won't it be "convenient" if their computer hard drives "crashed" as well, supposedly deleting the exact same time period? If something like that happens, then it will be incontrovertible proof that "the fix is in". (As if all of this isn't enough already.)
This stinks to high heaven. And it is not "tin-foil hat" to say so. 

[Logan Darklighter]

By the way, do feel free to try the "my hard drive failed" excuse if the IRS asks you to produce any document more than six months old if you think they'll extend to you the same benefit of the doubt that they give to Lois Lerner et al. 

[khagler]

Dartz wrote:

---

Anyway, I thought ratfucking was a traditional part of US politics. Using the organs of power to legally surpress your opponents by gerrmandering, redistrcting and otherwise making life as hard for them as possible. The difference between 'acceptable politics' and 'impeachable offence' is who's on the receiving end it seems.

It is indeed, and there's a long history of whichever faction is in power using the IRS to attack their political opponents.

[robkelk]

Logan, saying "carte blanche" or "drop in the bucket" is not proof for anything.

And I'd be extremely surprised of the IRS asked me to produce anything. Revenue Canada, OTOH...
--
Rob Kelk
"Governments have no right to question the loyalty of those who oppose
them. Adversaries remain citizens of the same state, common subjects of
the same sovereign, servants of the same law."

- Michael Ignatieff, addressing Stanford University in 2012

[Bob Schroeck]

khagler wrote:

---

Dartz wrote:

---

Anyway, I thought ratfucking was a traditional part of US politics. Using the organs of power to legally surpress your opponents by gerrmandering, redistrcting and otherwise making life as hard for them as possible. The difference between 'acceptable politics' and 'impeachable offence' is who's on the receiving end it seems.

It is indeed, and there's a long history of whichever faction is in power using the IRS to attack their political opponents.

It's a given that government is corrupt.  The only question is whether that corruption benefits you and your causes, or the other guy's.
-- Bob
---------
Then the horns kicked in...
...and my shoes began to squeak.

[RMH999]

Ah yes - I wonder if the IRS is using the same systems that lost the 22 million Bush Whitehouse emails...

CNet article

Washington Post article

Just to refresh your memories.
If we knew what it was we were doing, it would not be called research, would it?
- Albert Einstein

[Logan Darklighter]

Amazing that the IRS doesn't have a proper archive system for its emails. But that would take such enormous amounts of taxpayer money that it's a cost-saving measure that they don't. And we should thank them for saving money! 
Uh yeah... about that... 
Reason: The IRS had a contract with an Email back-up company called Sonasoft. In 2009, Sonasoft even sent out a Tweet advertising its work for the IRS.[img]http://cloudfront-media.reason.com/mc/psuderman/2014_06/sonasoft-IRS-tweet.png?h=386&w=650]

More from Reason: 

The Internal Revenue Service (IRS) said it can’t provide emails sent between 2009 and 2011 that were requested by congressional investigators because of hard drive crashes.
…
The IRS had a contract with email backup service vendor Sonasoft starting in 2005, according to FedSpending.org, which lists the contract as being for “automatic data processing services.” Sonasoft’s motto is “email archiving done right,” and the company lists the IRS as a customer.
…
...the company advertises its email archiving solution as “ideal for small and medium businesses, government agencies, school districts, nonprofit organizations using Microsoft’s Exchange Server.” And a document posted on its website describing its services says that its system “archives all email content and so reduces the risk of non-compliance with legal, regulatory and other obligations to preserve critical business content.”

The contract was cancelled a few weeks after Lerner’s computer allegedly crashed and just before other IRS officials’ computers are said to have crashed. 

My. How convenient. 

[Logan Darklighter]

For the resident IT folks, here's a synopsis of the work from a contract Sonasoft had with the IRS:
"The software initiates the real time replication of mail box changes and commences the backup of mailboxes via existing IRS servers. In addition, the software package includes the ability to rapidly switch to a disaster recovery site from a remote location. The software platform allows the OCC [IRS-Office of Chief Counsel’s] to recover customer e-mail in the event of a catastrophic Exchange server failure. Sonasoft Corporation has customized their commercial off the shelf software to operate within the OCC. Sonasoft is the sole proprietor of the software and software maintenance."http://www.fbo.gov/?s=opportunity&mode ... e&_cview=1
Sounds like files would be backed up redundantly to me.

[Dartz]

The place I used to intern with, they had a server that'd been in use since 1999. Almost constant use. It basically carried the weight of the entire place. All the corporate files were on it, including the accounts, .pst backups and account backups in case any of the company laptops - which dated to 2007/8 - failed on the road. Even then, the server had it's drives mirrored in RAID 1. I managed to convince them to make it properly redundant by having a simple automatic upload of the latest backup to spare space on our website host in case the server fell over, and erase the last one because space on the host was that tight and there was years of data in there.

I set it to do that late on Sundays, because it was the same place where our website was hosted and I didn't want to have it either down over the entire weekend, or go slow when people were trying to access it. And while the hosting agreement didn't specifically forbid us doing it , I didn't want to give them a reason to forbid it either.

(Before anyone screams 'You idiot'. I'm not an IT Professional or sysadmin, I don't officially work in IT, have no formal training or certifications, and I only did this because even I knew that the current setup was one fuckup away from falling over. It was something I hoped might give us a fighting chance if the server totally up and died. out of the three employees, I was the IT-guy by virtue of having hacked a new feature onto the company website)

One of the drives on the server decided to fail over the weekend while everyone was out, on a Saturday I guess. It didn't fail in a way that dropped it offline. It kept 'working' even as the thing was screaming, streaming out corrupted data. It streamed corrupted data all over its mirror, munging the backups. The main OS remained online, as it was on a separate disk. The job I'd set to run automatically on Sunday promptly read the corrupted files from the damaged RAID array, and dutifully uploaded them to the host after erasing the old backup because we just didn't have the space to keep them.

You can probably fill in the rest of what happened on Monday morning. Some of the mails still existed on the local machines - but archives and client-lists were ripped apart, along with some manuals and documentation - but the company accounts were mangled beyond repair.

The business fell over not long after that. While it'd been failing for a while - hence the reason why the hardware hadn't been upgraded or replaced - that was the thing that just tipped it right over the edge and the Directors pulled the plug after lurching onwards for another short while.

So it is entirely possible for backups to puke. Or be incompatible with changed variants of software, or have been lost to password changes. Or any number of things. It doesn't matter how 'by the numbers' things are done - IT failures will always find some way to propagate and always find that one hole you just didn't think of. Especially if something is automated, and is overseen by a person who thinks that because it's automated - they don't need to pay attention to it. And 'redundancy' means it has more of a chance to spread and propagate and update before anyone catches it.

The point being of all this is that, no matter how 'redundant' backups are, this still doesn't prove a coverup.
________________________________
--m(^0^)m-- Wot, no sig?

[Logan Darklighter]

Let me put this as plainly and clearly as possible:
The hard drives of Lois Lerner and 6 other people whose email files were subpoenaed were the only hard drives to fail. They all failed in exactly the manner required to destroy evidence of wrongdoing! 
Seven hard drives all failing at approximately the same time and they all happen to be the ones that were subpoenaed for evidence. What do you think the odds are that this is NOT a cover-up and clear obstruction of justice? Ten thousand to one? A million to one? A billion to one? 

I'd call it at "astronomical" and leave it at that. Anyone who can't see the pattern of abuse and corruption is fooling themselves. 

[Dartz]

It's more because, given how obscenely polarised and truthinessfull American politics has become that I'm trying to assume the real truth is somewhere back from what's being presented. Because everything these days seems to have some form of agenda and furthering that agenda rather than actually trying to report what happened. It's hard to take anything at face value anymore.

And, even if it is Government ordered. It seems to just be the done thing anyway. It's only illegal when the other guy's doing it to you. Then you get the big chair and do it right back to the other guy. 'When the President does it, it's not illegal' has been a part of US politics since long before Nixon, it seems.

Maybe I'm just too used to a government that's usually well meaning, but utterly incompetent.
________________________________
--m(^0^)m-- Wot, no sig?

[Logan Darklighter]

If incompetent but essentially well-meaning is all the Irish government is, I'm extraordinarily tempted to move there.