The Dyn DNS DDOS Attack Oct 21

DYN is a DNS provider internet infrastructure company. It’s the name behind widely used DynDNS. It supports DNS for twitter, visa, github, mongo, netflix and several other big tech sites.

Doug Madory a researcher at DYN, presented a talk  on DDoS attacks in Dallas at a meeting of the North American Network Operators Group (NANOG) 68 – his was the last talk on Oct 19, wednesday. The talk discussed the attack on Krebs on Security last month and details other such attacks.

On Friday several sites serviced by DYN were attacked in a distributed denial-of-service (DDoS) attack.

The distributed denial-of-service (DDoS) attack involved malicious DNS lookup requests from tens of millions of IP addresses including a botnet on a large number of IoT devices infected with the Mirai malware, which is designed to brute force security on any IoT device. There are cameras involved with fixed passwords that are burned into the firmware, that cannot be changed.

The implications of IoT devices that are 1) unsecure 2) impossible to secure and 3) infected by malware and 4) controlled by a botnet that is controlled by malicious intent are made clearer with this attack.

Update on DDOS mitigation:  RFC 3882,  Configuring BGP to block Denial-of-Service attacks, discusses Remote Triggered Black Hole (RTBH) method, to configure certain routers to selectively stop malicious high volume traffic which is targeting a particular IP.  The target site is made inaccessible but the rest of the network service stays active. An example configuration is at https://networkengineering.stackexchange.com/questions/10857/use-bgp-to-defend-against-a-ddos-attack-originating-from-remote-as .  The improvement is to reduce the side-effects of such a delisting, to enable faster recovery when the attack is over.

 

Ethical considerations in Autonomous Vehicles

A recent talk discussed ethics for autonomous vehicles, as an optimization problem. There can be several imperatives for an AV which are all “correct”, yet be in conflict for an autonomous vehicle which relies on hard coded logic.

For example: Follow Traffic safety rules. Stick to the lane. Avoid obstacles. Save most human lives. Save passengers.

How can a vehicle prioritize these ? Instead of a case by case design, the proposal is to cast it in an ethics framework based on optimization of various ideals and constraints with weighted coefficients. Then test the outcomes.

The optimization equation looks to minimize ( path_tracking + steering + traffic_laws ) subject to constraints ( avoid_obstacles ). The equations produce different behaviour when the coefficients are changed.

Another consideration is the Vehicle intent: is it fully in control or can the human override it. This affects the software assumptions and system design.

The talk was presented by Sarah Thornton, PhD. Stanford. A related discussion on safety is  here : Who gets the blame when driverless cars crash ?.

Somewhat related is the idea of computer vision itself operating correctly. There can be adversarial inputs as discussed in the paper Intriguing properties of neural networks which discusses blind spots. Generative Adversarial Models are a way to improve the generalization capabilities of a network by pitting generative against discriminative models. The European Conference on Computer Vision starts today: http://www.eccv2016.org/main-conference/

 

Neural Network Training and Inferencing on Nvidia

Nvidia just announced the Tesla P40 and P4 cards for Neural network inferencing applications. A review is at http://www.anandtech.com/show/10675/nvidia-announces-tesla-p40-tesla-p4. Comparing it to the Tesla P100 released earlier this year, the P40 is targeted to inferencing applications. Whereas the P100 was targeted to more demanding training phase of neural networks. P4o comes with the TensorRT (real time) library for fast inferencing (e.g. real time detection of objects).

Some of the best solutions of hard problems in machine learning come from neural networks, whether in computer vision, voice recognition, games such as Go and other domains. Nvidia and other hardware kits are accelerating AI applications with these releases.

What happens if the neural network draws a bad inference, in a critical AI application ? Bad inferences have been discussed in the literature, for example in the paper: Intriguing properties of neural networks.

There are ways to minimize bad inferences in the training phase, but not foolproof – in fact the paper above mentions that bad inferences are low probabalility yet dense.

Level 5 autonomous driving is where the vehicle can handle unknown terrain. Most current systems are targeting Level 2 or 3 autonomy. The Tesla Model S’ Autopilot is Level 2.

An answer is to pair it with a regular program that checks for certain safety constraints. This would make it safer, but this alone is likely insufficient either for achieving Level 5 operations, or for providing safely for them.

Automotive and Process Safety Standards

ISO 26262 is a standard for Automotive Electric/Electronic Systems safety, that is adopted by car manufacturers. Its V shape consists of two legs, the first comprising definition, analysis, design, architectural design, development and implementation. The second leg consists of verification and validation of the software, starting from unit tests to functional tests, safety tests and system-wide tests. Model based design is used to reduce the complexity. These models are now fairly complex. Model based design is the one of the value adds that Mentor Graphics automotive kit provides is help with achieving compliance with this standard.

ISO 26262 is derived from its parent, the IEC 61508 standard, which is titled Functional Safety of Electrical/Electronic/Programmable Electronic Safety-related Systems. This parent standard has variants for safety of automotive, railway, nuclear, manufacturing processes (refineries, petrochemical, chemical, pharmaceutical, pulp and paper, and power) and machinery related electrical control systems. An associated, upcoming standard is the SAE J2980.

An excellent talk today by MIT fellow Ricardo Dematos discussed more comprehensive approaches to automotive safety. This is building up from his work with safety research at MIT, AWS IoT and our SyncBrake entry for V2V safety at TechCrunch Disrupt 2015.

Embedded Neural Nets

A key problem for embedded neural networks is reduction of size and power consumption.

The hardware on which the neural net runs on can be a dedicated chip, an FPGA, a GPU or a CPU. Each of these consumes about 10x the power of the previous choice. But in terms of upfront cost, the dedicated chip costs the highest, the CPU the lowest. An NVidia whitepaper compares GPU with CPU on speed and power consumption. (It discusses key  neural networks like AlexNet. The AlexNet was a breakthrough in 2012 showing a neural network to be superior to other image recognition approaches by a wide margin).

Reducing the size of the neural network also reduces its power consumption. For NN size reduction, pruning of the weak connections in the net was proposed in “Learning both Weights and Connections for Efficient Neural Networks” by Song Han and team at NVidia and Stanford. This achieved a roughly 10x reduction in network size without loss of accuracy. Further work in “Deep Compression” achieved a 35x reduction.

Today I attended a talk on SqueezeNet by Forrest Iandola. His team at Berkeley modified (squeezed) the original architecture, then applied the Deep Compression technique above to achieve a 461x size reduction over the original, to 0.5Mb. This makes it feasible for mobile applications. This paper also references the V.Badrinarayan’s work on SegNet – a different NN architecture, discussed in a talk earlier this year.

The Nervana acquisition by Intel earlier this year was for a low power GPU like SOC chip with very high memory bandwidth.

MICROS Point-of-Sale POS attacks

Oracle MICROS Point of Sale systems are again reported to be attacked. The support site of MICROS was infected by malware which was able to access usernames and passwords and send them to a remote server. This remote server was identified as one previously known to be used by ‘Carbanak’, a cybercrime group.

ZDNet reports that in the last year, dozens of machines at Starwood and Hilton hotels were impacted by malware, with the aim of poaching payment and card data, which can be used or sold on to the highest bidder. The attack on MICROS systems may be behind these.

An advisory by VISA here, discusses two previous malware threats attacking POS systems, Carbanak and MalumPOS.

This report by Symantec describes POS attacks as multi-stage attacks, with threats and mitigation strategies.

In August 2014, Department of Homeland Security had issued an advisory for ‘Backoff’ a POS malware, which affected a large number of systems in retailers including Target, PF Chang’s, Neiman Marcus, Michaels, Sally Beauty Supply and Goodwill Industries International.  PF Chang had a cyber insurance policy which covered direct damages, but not claims filed by Mastercard/Bank of America for credit card fraud reimbursements and reissuance via a “PCI DSS assessment”.

These trends are interesting for a few reasons. First, the recurrent attack are reason to accelerate the move to EMVs. Second, it gives rise to new architectures for payments. Third it draws attention to Blockchain technologies.

Verifone has a Secure Commerce Architecture which sends the payment data directly from the terminal system which receives the card, to the merchant (acquirer) bank, without touching the POS system (the windows computer handling invoicing). This reduces payment fraud and also makes certification for EMVs much easier.

sca_diagram

EMV stands for Europay, Mastercard, Visa. After the EMV deadline of Oct 1, 2015, the liability for credit card fraud shifted to whichever party is the least EMV-compliant in a fraudulent transaction. Automated fuel dispensers have until 2017 to make the shift to EMV. ATMs face two fraud liability shift dates: MasterCard’s in October 2016 and Visa’s in October 2017.

Because of EMV securing the terminals, there is predicted to be a rise in online payments fraud.  While counterfeit opportunity dwindles, the next top three types of credit card fraud, Account takeover, Card-not-present fraud, and (fake new) Application fraud are rising. The recent MICROS attack fits the pattern of attackers probing for and finding the weakest links in the payments chain.

A blackhat talk –https://www.blackhat.com/docs/us-14/materials/us-14-Zaichkowsky-Point-Of-Sale%20System-Architecture-And-Security.pdf

ASN.1 buffer overflow puts networks at risk

An ASN.1 vulnerability has been found in a security advisory of 7/18 here. It has to do with length bounds checking in the LTV triplet.  A fix is provided but updating it on a large number of GSM devices is not practical. “It could be triggered remotely without any authentication in scenarios where the vulnerable code receives and processes ASN.1 encoded data from untrusted sources, these may include communications between mobile devices and telecommunication network infrastructure nodes, communications between nodes in a carrier’s network or across carrier boundaries, or communication between mutually untrusted endpoints in a data network.”

A discussion at Arstechnica here, brings up a real exploit against GSM base station software, that operates below the application layer and so can be exploited against a  large number of devices.

A quote from the paper – “When GSM radio stacks were implemented, attacks against end devices were not much of a concern. Hence checks on messages arriving over the air interface were lax as long as the stack passed interoperability tests and certifications. Open-source solutions such as OpenBTS (Base Transceiver Station)allow anyone to run their own GSM network at a fraction of the cost of carrier-grade equipment, using a sim- ple and cheap software-defined radio. This development has made GSM security explorations possible for a significantly larger set of security researchers. Indeed, as the reader will see in the following, insufficient verification of input parameters transmitted over the air interface can lead to remotely exploitable memory corruptions in the baseband stack.”

The cellular baseband stack of most smartphones runs on a separate processor and is significantly less hardened, if at all. GSM does not provide mutual authentication, there is no protection against fake BTSs.

Bitcoin ransomware attacks – malware

Two ransomware attacks happened in March where the program encrypted files on computer systems in hospitals, in Kentucky and California, then demanded payment in bitcoin for access to the encryption key.

This month a similar attack happened on a hospital in Calgary.

Citrix recently reported a large percentage of enterprises are now buying bitcoin to protect against such an attack.

Backing up systems is important to recovery. Another type of countermeasure is VDI where the desktop is rendered from a remote VM where the information is protected.

Lexus bad-update details

A bad update on Lexus cars crashed its car entertainment system, affecting cars from California to Massachusetts. Details were reported  by SecurityLedger this week.

Users need to bring in their car to a Toyota/Lexus dealer to solve the problem via a forced reset and clear the bad data from the system. The bad behavior is due to incorrect handling of error data returned from third party web services.

“As more automakers embrace over-the-air software updates as a way to push out necessary fixes to vehicle owners, the prospect of unreliable and malicious updates causing real world disruptions has grown. In a March report to Congress (PDF), the U.S. Government Accountability Office (GAO) noted that modern vehicles feature many communications interfaces that are vulnerable to attack, but that measures to address those threats are likely years away, as automakers work to design more secure in-vehicle systems.” – SecurityLedger quote.

Updates and secure updates have not been a well-solved problem in the software world. A backup is usually recommended, but not always possible. Bringing the same solutions to cars and IoT seems like a bad idea.  The need for secure OTA auto updates to work has been noted, for e.g. at the “Five Star Automotive Cyber Safety Program” here and here. Yet it has not been a prominent part of the automotive manufacturers’ lexicon.

As we move towards autonomous cars and a corresponding increase in complexity, these problems would need to be solved in more elegant way.

ICSA Internet of Things Security Certification Requirements

ICSA recently announced an Internet of Things testing and certification program. It has six components (highlights in brackets) –

  1. cryptography (FIPS 140-2 crypto algos by default, secure PRNGs)
  2. communications (PKI auth, all traffic must be authorized)
  3. authentication (secure auth, protect auth data, no privilege escalation)
  4. physical security (tamper detection, defense, disable)
  5. platform security (secure boot, secure remote upgrade, DoS defense)
  6. alert/logging (log upgrades, attacks, tampering, admin access)

Their IoT security requirements framework is found here.

This is a great list. I think another dimension to think about is usability of the security – many products come with security options buried so deep in documentation or UI, that a regular user may not configure the device securely and leave it more open than intended – this has historically been true of a variety of webcams, SCADA systems, wifi routers and other devices.

WebServices Composition with AWS

Some interesting diagrams on composition of a device data processing pipeline with AWS are at –http://aws-de-media.s3.amazonaws.com/images/jmetzner_Hackday_Berlin.pdf

The services listed are:
Amazon Cognito: Identity and Security. Gets token with role for API access by a certain user.
Amazon Kinesis: Massive data ingestion. Uses token auth, but token signing can be a easiest.
AWS Lambda: Serverless Data Compute. Supposed to save on EC2 instance costs ( at the expense of lock-in ).
Amazon S3: Virtually unlimited storage. This is what really makes AWS tick.
Amazon Redshift: Petabyte-scale data analysis
It does not say what data goes to S3 and what data goes to the database.
On Redshift, here’s a comment from Nokia:
http://www.cio.com/article/2860383/data-warehousing/7-amazon-redshift-success-stories.html#slide3” where their volume of data “literally broke the database”, prompting them to look for more scalable solutions.
There is a tension between “servers” and “services”, which goes back to IAAS vs PAAS distinction. PAAS can be faster to develop with reduced focus on server maintenance. However the number of PAAS concepts to deal with is neither small nor particularly inviting, as instead of a single server, one now has to deal with multiple services, each has to be authenticated, priced,  guarded for possible misuse and each has the potential for surprises. A key to simplicity is how composable the services are.

Security Competition Open Sourced

Facebook made a Capture The Flag (CTF) cybersecurity competition open source and available this week  at https://github.com/facebook/fbctf .

There are several other CTF projects on github. I like that this approach to cybersecurity gets one thinking like an attacker. The problem is that the attack surface in highly connected systems is not obvious or easily modeled.

How about CTF competitions for IoT Security? There was one in March – http://www.wamda.com/memakersge/2016/04/challenges-possibilities-iot-big-data

capture_the_flah_competition_on_day_3_at_the_innovation_zone_courtesy_ftw

 

Reactive Microservices

I came across CQRS and attended a talk by James Roper on Lagom for micro services called “Rethinking REST” this week. The idea put forth in the talk was that REST services being synchronous are not ideal for microservices. Microservices should not be blocking , so something that emphasizes this async aspect would be preferable to REST.

How is this notion of async services different from a pub-sub model ? One way it goes beyond that is by proposing polyglot persistence. Different microservices should use their own persistence layer that is optimal – relational, nosql, timeseries, event log.

Lagom architecture is based on the book Reactive Services Architecture and that also suggests CQRS. The book proposes service isolation and that composition of systems with microservices should be done asynchronously via message passing.

A quote from the book on why bulkheads failed to save the Titanic:

“The Titanic did use bulkheads, but the walls that were supposed to isolate the compartments did not reach all the way up to the ceiling. So when 6 out of its 16 compartments were ripped open by the iceberg, the ship started to tilt and water spilled over from one compartment to the next, until all of the compartments were filled with water and the Titanic sank, killing 1500 people.”

The suggestion is that for higher availability there should be stricter isolation. The individual (micro)services may fail but the overall system should not be affected. Looking at it this way I think requires one to examine more closely the system design and its invariants.

Take CQRS as an example. In a query system, data is being read and not being modified; a query is a search operation on accumulated data – potentially a very large set with speed (availability) demands. Whereas in a command system, there is more real-time,  perhaps collaborative aspect which leaves most of the data untouched, but creates some new data which needs to be recorded (upload this image, send this message). Why should these two very different operations be served by the same backend ?

One of the systems that has done separation of concerns remarkably well is  AWS. Let’s take the separation of S3 and the database (say Dynamodb). Neither filesystem vendors nor database vendors came up with simple services to solve the problem of an app trying to upload an object and update a database that the object has been uploaded. The failure modes of each service are exposed to the client and the client is not forced to upload files to S3 via a server (bottleneck). S3 offers a read-write consistency for new uploads and eventually consistency for updates. Here the storage object, its universally addressable name, and its properties (backup, encryption, access, versioning, retention policies) are an invariant -a client does not need to fiddle with them apart from being assured of a certain level of service. One can call it an ingestion system.  More on that here.

The takeaway is that microservices should do one thing and do it very well, in a highly available, non-blocking manner. REST services can certainly be non-blocking, but they can also be blocking as described here, which is a problem.

A cornerstone of asynchronous services is messaging. Here’s a talk on Riak, a masterless database and messaging system, running on CloudFoundry, at GE – https://youtu.be/aYaz17qf7cE?t=2161 . It is followed by talks on microservices.