Let’s take a disaster scenario where a system loses its data-in-transit (i.e. not yet persisted) at a certain point in time. and some time after this point, a recovery process kicks in, which restores the system back to normal functioning.

Recovery Point Objective refers to the amount of tolerable data loss measured in time. It can be measured in time based on the fact that it is in-transit data of a certain max velocity, so bounding the time bounds the amount of data that can be lost. This time figure, the RPO, is used to determine how frequently the data must be persisted and replicated. An RPO of 10 minutes implies the data must be backed up every 10 minutes. If there’s a crash the system can be restored to a point not more than 10 minutes prior to the time of crash. RPO determines frequency of backups, snapshots or transaction logs.

Recovery Time Objective refers to the amount of time required to restore a system to normal behavior after a disaster has happened. This includes restoration of all infrastructure components that provide a service, not just the restoration of data.

Lower RPO/RTO is higher cost.

Matrix of RPO – high/low vs RTO – high/low can be used to categorize applications.

Low RPO, Low RTO. Critical online application like a storefront.

Low RPO, High RTO. Data sensitive application but not online, like analytics.

High RPO, Low RTO. Redundantly available data or no data. Compute clusters that are highly available.

High RPO, High RPO. Non-prod systems – dev/test/qa ?

Amount of acceptable data loss <= App (data?) criticality.
One can expect a pyramid of apps – large number with less criticality, small number with high criticality

Repeatability. Backup and recovery procedures. Must be written. Must be tested. Automation.

HA/DR spectrum of solutions:

• Backups, save transaction logs
• Snapshots
• Replication – synchronous, asynchronous
• Storage only vs in-memory as well. Application level crash consistency of backups.
• Multiple AZs
• Hybrid

Tech: S3 versioning and DDB streams, Global tables.

Rules of thumb:

• test full recovery regularly, at least once an year.
• backup, backup, backup
• 3-2-1 Rule. https://us-cert.cisa.gov/sites/default/files/publications/data_backup_options.pdf
• Keep at least 3 copies of data in at least 2 media types, and at least one off-site backup.

Related terms: RPA and RTA

3 types of disasters.

• Natural disaster – e.g. floods, earthquakes, fire
• Technical failure – e.g. loss of power, cable pulled
• Human error – e.g. delete all files as admin

Replication – works for first two. Continuous snapshots/backup/versioning – for the last one. Replication will just delete the data on both sides. Need the ability to go back in time and restore data.

Cost – how to optimize cost of infrastructure and its maintenance.

Which region to choose ? Key considerations: What types of disasters are the concern (Risk). How much proximity is needed to end-customers and to primary region (Performance). What’s the cost of the region (Cost) ?

Notes on a talk on the data platform for Threat detection and response at scale at Spark+AI Summit, 2018.

The threat detection problem, use-cases and scale.

• It’s important to focus on and build the data platform first else one can get siloed into narrow set of addressable use-cases.
• we want to detect attacks,
• contextualize the attacks
• determine root cause of an attack,
• determine what the scope of the incident might be
• determine what we need to contain it
• Diverse threats require diverse data sets
• the Threat signal can be concentrated or spread in time
• Keylines visualization library is used to build a visualization of detection, contextualization, containment

Streaming is a simple pattern that takes us very far for detection

• Streams are left-joined with context and filtered or inner-joined with indicators
• Can do a lot with this but not everything
• Graphs are key. Graphs at scale are super hard.

Enabling triage and containment with search and query

• to triage the detection, it comes down to search and query.
• ETM does 3.5million records/sec. 100TB of data a day. 300B events a day.
• 11 trillion rows, 0.5PB of data.

Ingestion architecture – tries to solve all these problems and balance issues.

• data comes into s3 in a consistent json wrapper
• there’s a single ETL job that takes all the data and writes it into a single staging table which is partitioned by date and event-type, has a long retention
• table is optimized to stream new data in and stream data out of, but can be queried as well. you can actually go and query it using sql function
• highest value data – we write parsers, we have discrete parsing streams and put them into a common schema and put it into separate delta tables. well parsed, well structured.
• use optimizations from delta, z-odering.. to index over columns that are common predicates. search by IP address, domain names – those are what we order by
• indexing and z-ordering – take advantage of data skipping
• sometimes we parser code gets messed up.
• single staging table.. is great . we just let the fixed parser run forward, we have all the data corrected, then we are back-corrected. don’t have to repackage code and run as a batch job. we literally just fix code and run it in the same model that’s it.
• off of these refined tables or parsed data sets, this is where the detection comes in.
• we have a number of detection streams in batches, that do the logic and analysis. facet-faced or statistical.
• alerts that come out of this go to their own alert table. goes to delta again. long retection, consistent schema. another streaming job then does de-duplication and whitelisting and writes out alerts to our alert management system. we durably store all the alerts, whether or not de-duped/whitelisted
• allows us to go back and fix things if things are not quite correct, accidentally.
• all this gives us operational sanity, and a nice feedback loop

Thanks to z-ordering, it can go from scanning 500TB to 36TB.

• average case is searching over weeks or months. it makes it usable for ad-hoc refinements.
• simple, unified platform.

Michael: Demo on interactive queries over months of data

• first attempt is sql SELECT on raw data. takes too long, cancelled. second attempt uses HMS, still too long, cancelled. why is this so hard ?
• observation: every data problem is actually two problems 1) data engineering and 2) data science. most projects fail on step 1.
• doing it with delta – the following command takes 17s and fires off spark job to put the data in a common schema.

CREATE TABLE connections USING delta AS SELECT * from json.'/data/connections'

then

SELECT * FROM connections WHERE dest_port = 666

this is great to query the historical data quickly.. however batch alone is not going to cut it as we may have attacks going on right now. but delta plugs into streaming as well:

INSERT INTO connections SELECT * from kafkaStream

Now we’ve Indexed both batch and streaming data.

We can run a python netviz command to visualize the connections.

—

Here’s a paper on the Databricks Delta Lake approach – https://databricks.com/wp-content/uploads/2020/08/p975-armbrust.pdf .

A few interesting ideas from AWS Builders Library –

• design APIs to be safe for retries or idempotent for resiliency
• shuffle sharding to distribute load as system scales
• Amazon’s approach to failing successfully.

Toyota’s Five-why’s approach to root cause a problem – is good but not enough to find all other root causes that might also cause a problem.

Couple great talks on serverless

• https://www.youtube.com/watch?v=dzU_WjobaRA

• https://www.youtube.com/watch?v=Fup5vHEvU50

Smithy is an Apache-2.0 licensed, protocol-agnostic IDL for defining APIs, generating clients, servers and documentation. https://github.com/awslabs/smithy

Iptables is a Linux-based firewall utility that allows system administrators to configure rules and filters for network traffic. It works by examining each packet that passes through the system and making decisions based on rules defined by the administrator.

In the context of iptables, tables are organizational units that contain chains and rules. There are five built-in tables in iptables:

1. filter – This is the default table and is used for packet filtering.
2. nat – This table is used for network address translation (NAT).
3. mangle – This table is used for specialized packet alteration.
4. raw – This table is used for configuring exemptions from connection tracking in combination with the NOTRACK target.
5. security – This table is used for Mandatory Access Control (MAC) networking rules.

In iptables, a chain is a sequence of rules that are applied to each packet passing through a particular table. Each table contains several built-in chains, and these chains can also have user-defined chains.

Each table contains several built-in chains and can also have user-defined chains. The chains contain rules that are used to determine what happens to packets as they pass through the firewall.

The name “chain” comes from the way that iptables processes packets. When a packet arrives at the firewall, it is first matched against the rules in the PREROUTING chain. If the packet matches a rule in that chain, it is processed according to the action specified in the rule. If the packet is not matched by any rules in the PREROUTING chain, it is then passed to the INPUT chain, where it is again matched against each rule.

This process continues through the FORWARD, OUTPUT, and POSTROUTING chains until the packet is either accepted, rejected, or dropped. Because each chain is a sequence of rules that are processed in order, it is called a “chain.”

In iptables, there are five types of chains. Not all of the five chains are present in all the tables. The PREROUTING, INPUT, and FORWARD chains are present in the “filter” and “raw” tables. The OUTPUT and POSTROUTING chains are present in the “filter”, “nat”, and “mangle” tables.

The “security” table, which is used for Mandatory Access Control (MAC) networking rules, does not have the PREROUTING, POSTROUTING, or OUTPUT chains. Instead, it has inbound and outbound chains for each network interface.

Each of these chains has a default policy that can be set to either ACCEPT, DROP, or REJECT. ACCEPT means that the packet is allowed through, DROP means that the packet is silently discarded, and REJECT means that the packet is discarded and the sender is notified.

System administrators can create custom rules that match specific criteria, such as the source or destination IP address, the protocol used, or the port number.

“mangle” is a table that is used to alter or mark packets in various ways. The mangle table provides a way to modify packet headers in ways that other tables cannot, such as changing the Time To Live (TTL) value or the Type of Service (TOS) field.

Remote Desktop Protocol (RDP) has its own vulnerabilities and needs to be appropriately secured.

https://mmishou.wordpress.com/2010/04/13/passive-dns-mining-from-pcap-with-dpkt-python/

Ransomware-spreading hackers sneak in through RDP

https://www.n00py.io/2018/08/bypassing-duo-two-factor-authentication-fail-open/

I was struck by the large number of vendors offering visibility as a key selling point. Into various types of network traffic. Network monitors, industrial network gateways, SSL inspection, traffic in VM farms, between containers, and on mobile.

More expected are the vendors offering reduced visibility – via encryption, TPMs, Secure Enclaves, mobile remote desktop, one way links, encrypted storage in the cloud etc. The variety of these solutions is also remarkable.

Neil Degrasse’s closing talk was so different yet strangely related to these topics, the universe slowly making itself visible to us through light from distant places, with insights from physicists and mathematics building up to the experiments that recently confirmed gravitational waves – making the invisible, visible. . This tenuous connection between security and physics left me misty eyed. What an amazing time to be living in.

Software Defined Networking seeks to centralize control of a large network. The abstractions around computer networking evolved from the connecting nodes via switches, to applications that run on top with the OSI model, to the controllers that manage the network. The controller abstraction was relatively weak – this had been the domain of telcos and ISPs, and as the networks of software intensive companies like Google approached the size of telco networks, they moved to reinvent the controller stack.  Traffic engineering and security which were done in disparate regions were attempted to be centralized in order to better achieve economies of scale. Google adopted openflow for this, developed by Nicira, which was soon after acquired by VMWare; Cisco internal discussions concluded that such a centralization wave would reduce Cisco revenues in half, so they spun out Insieme networks for SDN capabilities and quickly acquired it back. This has morphed into the APIC offering.

The centralization wave is a bit at odds with the security and resilience of networks because of their inherent distributed and heterogenous nature. Distributed systems provide availability, part of the security CIA triad, and for many systems availability trumps security. The centralized controllers would become attractive targets for compromise. This is despite the intention of SDN, as envisioned by Nicira founder M. Casado, to have security as its cornerstone as described here. Casado’s problem statement is interesting: “We don’t have a ubiquitous and horizontal security layer that provides both context and isolation. Where do we normally put security controls? We put it in one of two places. We might put it in the physical infrastructure, which is great because you have isolation. If I have ACLs [access control lists] or a firewall or an IDS [intrusion detection system], I put it in a separate box and I put it away from the applications so that the attack surface is pretty small and it’s totally isolated… you have isolation, but you have no context. ..  Another place we put security is in the end host, an application or operating system. This suffers from the opposite problem. You have all the context that you need — applications, users, the data being accessed — but you have absolutely no isolation.” The centralization imperative comes from the need to isolate and minimize the trusted computing base.

In the short term, there may be some advantage to be gained by complexity reduction through centralized administration, but the recommendation of dumb switches that respond to a tightly controlled central brain, go against the tenets of compartmentalization of risk and if such networks are put into practice widely they can result in failures that are catastrophic instead of isolated.

What the goal should be is a distributed system which is also responsive.

Cgroups limit how much you can do. Namespaces limit how much you can see.

Linux containers are based on cgroups and namespaces and can be privileged or unprivileged. A privileged container is one without the “User Namespace” implying it has direct visibility into all users of the underlying host. The User Namespace allows remapping the user identities in the container so even if the process thinks it is running as root, it is not.

Using cgroups to limit fork bombs from Jessie’s talk:

$ sudo su
# echo 2 > /sys/fs/cgroup/pids/parent/pids.max
# echo $$ > /sys/fs/cgroup/pids/parent/cgroups.procs  // put current pid
# cat /sys/fs/cgroup/pids/parent/pids.current
2
# (echo "foobar" | cat )
bash: for retry: No child processes

Link to the 122 page paper on linux containers security here.  Includes this quote on linux kernel attacks.

“Kernel vulnerabilities can take various forms, from information leaks and Denial of Service (DoS) risks to privilege escalation and arbitrary code execution. Of the roughly 400 Linux system calls, a number have contained privilege escalation vulnerabilities, as recently as of 2016 with keyctl(2). Over the years this included, but is not limited to: futex(2), vmsplice(2), mremap(2), unmap(2), do_brk(2), splice(2), and modify_ldt(2). In addition to system calls, old or obscure networking code including but not limited
to SCTP, IPX, ATM, AppleTalk, X.25, DECNet, CANBUS, Econet and NETLINK has contributed to a great number of privilege escalation vulnerabilities through various use cases or socket options. Finally, the “perf” subsystem, used for performance monitoring, has historically contained a number of issues, such as perf_swevent_init (CVE-2013-2094).”

Which makes the case for seccomp, as containers both privileged and unprivileged can lead to bad things –

““Containers will always (by design) share the same kernel as the host. Therefore, any vulnerabilities in the kernel interface, unless the container is forbidden the use of that interface (i.e. using seccomp)”- LXC Security Documentation by Serge Hallyn, Canonical”

The paper has several links on restricting access, including grsecurity, SELinux, App Armor and firejail. A brief comparison of the first three is here. SELinux has a powerful access control mechanism – it attaches labels to all files, processes and objects; however it is complex and often people end up making things too permissive, instead of taking advantage of available controls.  AppArmor works by labeling  files by pathname and applying policies to the pathname – it is recommended with SUSE/OpenSUSE, not CentOS.  Grsecurity policies are described here, its ACLs support process–based resource restrictions, including memory/cpu/files open, etc.

A recent talk discussed ethics for autonomous vehicles, as an optimization problem. There can be several imperatives for an AV which are all “correct”, yet be in conflict for an autonomous vehicle which relies on hard coded logic.

For example: Follow Traffic safety rules. Stick to the lane. Avoid obstacles. Save most human lives. Save passengers.

How can a vehicle prioritize these ? Instead of a case by case design, the proposal is to cast it in an ethics framework based on optimization of various ideals and constraints with weighted coefficients. Then test the outcomes.

The optimization equation looks to minimize ( path_tracking + steering + traffic_laws ) subject to constraints ( avoid_obstacles ). The equations produce different behaviour when the coefficients are changed.

Another consideration is the Vehicle intent: is it fully in control or can the human override it. This affects the software assumptions and system design.

The talk was presented by Sarah Thornton, PhD. Stanford. A related discussion on safety is  here : Who gets the blame when driverless cars crash ?.

Somewhat related is the idea of computer vision itself operating correctly. There can be adversarial inputs as discussed in the paper Intriguing properties of neural networks which discusses blind spots. Generative Adversarial Models are a way to improve the generalization capabilities of a network by pitting generative against discriminative models. The European Conference on Computer Vision starts today: http://www.eccv2016.org/main-conference/

ISO 26262 is a standard for Automotive Electric/Electronic Systems safety, that is adopted by car manufacturers. Its V shape consists of two legs, the first comprising definition, analysis, design, architectural design, development and implementation. The second leg consists of verification and validation of the software, starting from unit tests to functional tests, safety tests and system-wide tests. Model based design is used to reduce the complexity. These models are now fairly complex. Model based design is the one of the value adds that Mentor Graphics automotive kit provides is help with achieving compliance with this standard.

ISO 26262 is derived from its parent, the IEC 61508 standard, which is titled Functional Safety of Electrical/Electronic/Programmable Electronic Safety-related Systems. This parent standard has variants for safety of automotive, railway, nuclear, manufacturing processes (refineries, petrochemical, chemical, pharmaceutical, pulp and paper, and power) and machinery related electrical control systems. An associated, upcoming standard is the SAE J2980.

An excellent talk today by MIT fellow Ricardo Dematos discussed more comprehensive approaches to automotive safety. This is building up from his work with safety research at MIT, AWS IoT and our SyncBrake entry for V2V safety at TechCrunch Disrupt 2015.