Tuesday, September 13, 2016

Bright Talk Series on HPC system

https://www.brighttalk.com/mybrighttalk/knowledge-feed

Bright Talk Series on HPC system





    • Posted by at No comments:

    Monday, July 25, 2016

    Endian Firewall for The Community

    Short description

    The Endian Firewall is a turnkey Linux Security Distribution, which is an independent, unified security management solution. The Endian Firewall is based on a hardened Linux operating system. The system is installed on a PC using a boot CD and can be operated without a monitor through its online interface or via a keyboard in a command-line fashion. The server can be configured via a web interface or via the serial interface.
    The main task of Endian Firewall is a gateway, router and firewall, and can act as a proxy for web, email, FTP, SIP and DNS. Up to four different networks (dependent on the number of network cards installed in the host PC) are normally managed.Networks are configured through the web interface. With Endian these are differentiated by their color coding:
    • Red Network: connection to the insecure Internet.
    • Green Network: Secure intranet e.g. file server.
    • Orange Network: Part Safe Demilitarized Zone (DMZ) . This includes devices that operate their own server and must be accessible over the Internet, such as Web or FTP servers.
    • Blue Network: Secure wireless part, here on wireless devices can be connected. Thus, they are separated from the green network, which increases its security.
    The Endian Firewall includes support for load balancing, which means you can add another connection to the Internet from the red network; Endian Firewall then distributes the network load on both interfaces.

    License

    Behind the Endian Firewall is the Italian Endian Spa from Appiano, South Tyrol and a community of volunteer developers and helpers. The license model of Endian provides a commercial version and a free version:
    • The commercial version can be purchased either as a standalone software (the product is called Endian or simply Endian UTM Software) in order to install them on their own PCs, as well as in the form of finished Out of the Box - firewalls, with special hardware on which the software is preinstalled. There are currently seven hardware variants with different performance and for different network sizes: Mini 25, Mercury 50, Mercury 100, Macro 250, Macro 500, Macro 1000 and Macro 2500.
    • Both the free Endian Firewall Community and the commercial Endian UTM Software are tailored for installation on your own hardware. The Endian Firewall Community is licensed under the GPL and is therefore open source software that can be downloaded for free. The community version does not include support and not all the features of the commercial version are available.

    Featured

    The current version includes the following key features:

    Gateway

    • Ethernet support
    • Load Balancing
    • Traffic Shaping / Quality of Service
    • Multiple uplinks
    • Uplink failover

    Firewall & Security

    • Firewall (both directions)
    • Demilitarized Zone
    • Intrusion Detection System / Intrusion Prevention System
    • Web-, FTP- and E-Mail-antivirus
    • Antispam
    • Content Filter
    • HTTPS- Web interface
    • SSH- Access and Forwarding
    • Scheduler for automated backups

    Server Services

    User Management

    • Local
    • RADIUS
    • LDAP
    • Active Directory
    • NTLM Single Sign-On
    • User or group as HTTP proxy content filter rules

    Logging & Monitoring

    • Visualized Live Log Viewer (AJAX based), see figure "The web interface of Endian Firewall"
    • Log the activities and the stress of network and hardware
    • Connection statistics
    • Forwarding possibility of logs to an external syslog server
    • ntopng integration
    • Event-based notifications by e-mail

    Others

    • Support Software-RAID
    Posted by at 1 comment:

    Friday, July 15, 2016

    The Future of IPtables: NFtables

    What is nftables?

    nftables is the project that aims to replace the existing {ip,ip6,arp,eb}tables framework. Basically, this project provides a new packet filtering framework, a new userspace utility and also a compatibility layer for {ip,ip6}tables. nftables is built upon the building blocks of the Netfilter infrastructure such as the existing hooks, the connection tracking system, the userspace queueing component and the logging subsystem.

    What is the status of nftables?

    It is currently under development.

    Running nftables

    You require the following software in order to run the new nft userspace utility:
    • nftables's Linux kernel tree
    • libmnl: the minimalistic Netlink library
    • libnftnl: the user-space library for low-level interaction with nftables Netlink's API over libmnl.
    • nftables userspace utility
    This new utility uses a new syntax that is different from {ip,ip6,eb,arp}tables.

    Running {ip,ip6}tables compatibility

    The nftables project provides a backward compatibility layer that allows you run iptables/ip6tables (using the same syntax) over the nftables infrastructure:
    • nftables's Linux kernel tree
    • libmnl: the minimalistic Netlink library
    • libnftnl: the user-space library for low-level interaction with nftables Netlink's API over libmnl.
    • iptables-nftables: the user-space utility that provides the xtables command line utility to add rule using {ip,ip6}tables syntax.

    Main Features

    • Pseudo-state machine in kernel-space: the userspace utility nftables interprets the rule-set provided by the user (using a new syntax), it compiles it into the pseudo-state machine bytecode and then it transfers it to the kernel via the nftables Netlink's API. Roughly, the idea behind nftables is similar to the Berkeley Packet Filters (BPF).
    • Fast lookups through performance data structures: The new syntax allows you to arrange you rule-set in a very performance way contrary to purely linear-list based filtering policies. Nftables allows you to use set-based action mappings, ie. for a matching element in the set, issue the action specified by the user.
    • Reduce the amount of code in kernel-space. You can express the packet selectors for all existing protocols using the instruction-set provided by the nftables pseudo-state machine. That means that we do not need a specific extension in kernel-space for each protocol that you want to support. As a side effect, you are likely not need to upgrade your kernel to obtain new features as it has been designed to keep most of the logic in user-space.
    • Unified interface to replace iptables/ip6tables/arptables/ebtables utilities. Thus, we will be able to fully get rid of all the existing code replication in kernel and user-space.

    Git trees

    Documentation

    You can check the nftables HOWTO documentation.
    Posted by at No comments:

    Tuesday, May 31, 2016

    OpenHPC Project


    OpenHPC 1.1   (18 April 2016)

    Binary downloads are presently available in the form of RPMs. These RPMs are organized into repositories that can be accessed via standard package manager utilities (e.g. yum, zypper). OpenHPC provides builds that are compatible and tested against CentOS 7.2 as well as SUSE Linux Enterprise Server 12 SP1. A typical deployment on a new system will begin with the installation of the base operating system on a chosen master host identified as the system management server (SMS), followed by enabling access to a compatible OpenHPC repository.
    The OpenHPC repository is created and maintained using a dedicated instance of the Open Build Service (OBS) that is available here. In addition to serving as the build server, this OBS instance also provides an RPM repository. You can scan the RPM packages that are available via this repository by browsing the x86_64/ and noarch/ subdirectories for the 1.1 release at: http://build.openhpc.community/OpenHPC:/1.1.
    To get started, you can enable an OpenHPC repository locally through installation of an ohpc-release RPM which includes gpg keys for package signing and defines the URL locations for [base] and [update] package repositories. A copy of the ohpc-release file is available for download here:
    Alternatively, you can use the package manager to install the RPM directly from the network as in the following examples:
    # yum install https://github.com/openhpc/ohpc/releases/download/v1.1.GA/ohpc-release-centos7.2-1.1-1.x86_64.rpm
    or
    # zypper in https://github.com/openhpc/ohpc/releases/download/v1.1.GA/ohpc-release-sles12sp1-1.1-1.x86_64.rpm

    Install Recipe(s)

    To aid in the installation of OpenHPC packaged components,  a companion installation recipe is available. This can be obtained via installation of the docs-ohpc RPM after the OpenHPC repository has been enabled locally. Alternatively, copies of the documentation are also provided below:
    The intent of the guide is to present a simple cluster installation procedure using components from the OpenHPC software stack. The documentation is intended to be reasonably generic, but uses the underlying motivation of a small, stateless cluster installation to define a step-by-step process. Several optional customizations are included and the intent is that these collective instructions can be modified as needed for local site use cases.  Please consult the install guide for more detail and discussion regarding a companion template install script.


     ***********************************

    About OpenHPC

    Mission

    OpenHPC is a Linux Foundation Collaborative Project whose mission is to provide an integrated collection of HPC-centric components that can be used to provide full-featured reference HPC software stacks. Provided components should range across the entire HPC software ecosystem including provisioning and system administration tools, resource management, I/O services, development tools, numerical libraries, and performance analysis tools.
    To support this mission, the following sections highlight elements of the community vision and key values.

    Vision

    • to provide a collection of pre-packaged binary components that, when combined with a supported base operating system (BoS), can be used to install and manage HPC systems throughout its lifecycle to provide a stable, feature-rich development and runtime environment
    • to provide HPC-centric packages that are either absent or have unacceptable lag time from leading Linux distro providers
    • to support new hardware offerings from vendors in a timely fashion
    • to provide distribution/installation mechanisms for leading research groups releasing open-source software
    • to allow both open-source and proprietary software vendors to focus efforts on innovation
    • to allow and promote multiple system configuration recipes that leverage community reference designs
    • to foster development of defined interfaces between supported components that allows for simple component replacement and customization

    Posted by at No comments:

    Lustre Takes Aim at Enterprise Security Requirements

    secure-file-shutterstock_147252422
    “Security in Lustre reflects its roots as a high performance computing (HPC) file system,” said John Hammond of Intel’s High Performance Data Division. In traditional HPC deployments, Lustre resides in a physically secure and logically isolated environment, dependent on strict network access controls, managed by large and siloed teams focused on what’s happening with the systems, and accessed by users already holding appropriate clearances to log on and use the resources. According to Hammond, “Lustre assumes that if a client could connect, it would connect.” Of course, this presents security risks for today’s enterprise environments.
    “In the enterprise, computing resources are often connected to wider, shared networks—including the Internet—that may be directly or indirectly accessible to a wide range of users with various privileges,” added Andreas Dilger, principal engineer for Intel’s Lustre development team. “Data center management teams are usually smaller, spread thin in their diverse responsibilities, and used to dealing with more standard security infrastructures and mechanisms that have protections inherent in the file system.” Thus, as Lustre has expanded into more corporate environments, its users have requested deeper security in the file system. Now available, version 2.8 of the community release and 3.0 of the Intel Enterprise Edition for Lustre software (Intel EE for Lustre software) have added three critical security features in Lustre: client and user authentication, policy-based file access restrictions, and encryption.
    Kerberos Authentication and Encryption is Back
    Kerberos has long been a part of Lustre, but until a couple of years ago, it had been largely unused for various reasons. Community developers revived Kerberos to provide fundamental security features needed by enterprise IT departments. Today, Kerberos version 5 in Lustre can authenticate individual clients, users, and the Meta data servers (MDSs) to allow clients to mount the file system and give users access to the files. Authentication is mutual—client to server and server to client—using standard Linux user credentials and Kerberos-generated keys.
    Kerberos in Lustre
    Kerberos in Lustre
    Kerberos in Lustre is designed to authenticate users with the MDSs only. Authenticating every request and authenticating the Object Storage Servers is beyond the current implementation of Kerberos on Lustre.
    Additionally, Kerberos provides encryption for information sent over the network, whether it be keys, credentials, or data. The technology has several encryption algorithms at its disposal, and Intel processors integrate hardware acceleration for the algorithms used in Lustre. Performance testing in 2015 by Sebastien Buisson of OpenSFS member Data Direct Networks revealed a very modest impact with authentication, as seen in his presentation (http://wiki.lustre.org/images/e/ec/Lustre-and-Kerberos_Buisson.pdf) at the 2015 Lustre User Group (LUG). When encryption was enabled, the impact was much more significant, but the system still delivered 500 to 700 MB/sec, which is still very fast when compared to WAN performance where this would typically be needed. Thus Kerberos gives important security capabilities for critical workloads in some industries, such as health sciences, where data security in transit is often a requirement.
    Mandatory Access Controls with SELinux
    In an HPC environment, file owners are used to controlling their file access permissions. But organizations with shared sensitive data on an open network often need a greater level of control by some form of policy management. In Lustre, this is implemented with mandatory access controls using SELinux on the client.
    SELinux is part of many Linux distributions. SELinux takes file access control out of user space and puts it in kernel space. It labels files and, using type-enforcing mode, enforces security policies on the client, based on the process context. The label is stored in the file’s extended attributes (xattr), so that when the client fetches the label and caches it, SELinux blocks file access to unauthorized clients and grants access to authorized clients.
    The way SELinux is implemented in Lustre 2.8 Community edition and 3.0 Intel EE for Lustre software, the SELinux context is not passed to the Meta data targets. It is not enforced on the server. However, using Kerberos, Lustre authenticates a client and user to an MDS, establishing a trust relationship. SELinux, then, enforces that the client has access only to allowed files. Additionally, there are other mechanisms, which are often implemented in IT departments to mitigate lack of SELinux context on the server, including audits, restricting clients that can connect to trusted clients, monitoring configurations to ensure only correctly configured clients are available to connect, and scanning.
    SELinux is required in some environments, where it is certified with code suites using approved tests. Many certification suites and tests are not public, and they are tied to the whole security stack, which gets certified together. The Lustre development community does not have access to the entire suite in any one environment. Thus, SELinux in Lustre was limited to fulfill a certain level of security within the file system, so it could be certified as part of the whole stack.
    More Security Choices Coming
    The Lustre developer community continues to enhance Lustre security with more choices in upcoming software releases.
    “Kerberos is a well-known authentication protocol and effective, but it is complex to set up, and it presents technical and political challenges in large organizations when needing to do cross-realm authentication,” said Malcolm Cowe, Lustre product manager for Intel’s High Performance Data Division. “Different sites are typically managed independently. Thus, different users from different administrative domains won’t necessarily have the same UID across the domains. There usually is no unified password file to map differing UIDs across systems to a single user.” But, Lustre uses numerical user IDs (UIDs), making cross-realm authentication impossible without some type of unifying identification.
    Indiana University is driving development of a solution that creates a unified identification map based on network IDs (NIDs) that translates the UID and group ID (GID) across realms to the NIDs. The code allows managers to configure ranges of NIDs and map them a certain way so users across domains can access different file systems. “Based on the NID map,” explained Dilger, “if a client with a UID in a remote domain wants to mount a file on a local system, mapping will translate the UID to the proper user on the local system.”
    “Besides being a way to simplify cross-realm authentication,” added Hammond, “having a unified identification scheme is a useful security tool. Managers can squash all non-authorized Lustre users in the map, like in NFS.” UID/GID mapping is expected to land in Lustre Community version 2.9.
    Where organizations don’t want to use Kerberos for authentication and encryption, code for node authentication only from Indiana University will offer a simpler alternative based on Shared Secret Keys (SSKs). The SSKs method does not authenticate users, and it does not require UID/GID mapping, since it is using SSKs to allow a node access to the file system, even a remote node. But, when it is combined with UID/GID mapping, it enables strong remote, cross-domain authentication and node blocking that does not involve the complexities of setting up Kerberos. The Indiana University solution provides encryption as well as authentication, and it is expected to land in Lustre 2.9.
    In enterprises, home directories are commonplace. Yet Lustre exposes the user to the entire namespace. Data Direct Networks has been working on code that will restrict view of the Lustre file system namespace at client mount time to only the subdirectory the client is accessing. “The code changes the FID of the subdirectory to the root of the client directory,” said Hammond. “The client will no longer see the root of the Lustre file tree.” Alone this does not provide adequate security, according to Hammond, but in an authenticated environment and using UID/GID mapping in wide networks, it restricts what subdirectories can be seen by clients. Additionally, subdirectory mounting can be containerized. Each container will mount as a Lustre client and be authenticated with its own credentials. Then, the subdirectory the container has access to can be mounted in the root of the container.
    Lustre Spring Cleaning—Ongoing Code Hardening to Improve Security
    Clean code is securer code. “We’ve been doing a lot of code cleanup in Lustre of late,” stated Hammond.
    “Lustre is 15 years old,” added Dilger. “It has evolved from the Linux 2.4 kernel through the latest kernels, so there’s a lot of code that had originally been needed for compatibility between different kernel versions. Compatibility layers, user space, prototype codes for different platforms that would never be fully implemented—there were lots of wrappers and abstraction layers that needed to be removed. So, we did some spring cleaning. And we continue the efforts.”
    Beyond Lustre 2.9
    According to Cowe, Lustre security has come a long way. “We’ve added authentication and encryption and mandatory access control support in version 2.8. There are more options coming in 2.9. And we are looking at end-to-end enforcement going forward.”
    Author’s note: This year’s Lustre User Group (LUG) included several presentations on security in Lustre. They are well worth reviewing for any Lustre administrator, user, or prospective user. They can be viewed at http://wiki.lustre.org/Lustre_User_Group_2016.
    Posted by at No comments:

    Wednesday, May 25, 2016

    Piping TAR to move files and directories around: one command line

    Move files and directories from one location to another via TAR and pipe:

     (cd dir1 && tar cvf - . ) | (cd dir2 && tar xvf -)


    ( cd SOURCEDIR && tar cf - . ) | (cd DESTDIR && tar xvpf - )

    where - stands for stdout.

     Neat?




    The obvious way to copy directories with tar is to write them onto a tape archive with relative pathnames -- then read back the tape and write it somewhere else on the disk. But tar can also write to a UNIX pipe -- and read from a pipe. This looks like:
    % reading-tar | writing-tar
    with one trick: the writing-tar process has a different current directory (the place where you want the copy made) than the reading-tar. To do that, run the writing-tar in a subshell.
    The argument(s) to the reading-tar can be directory(s) or file(s). Just be sure to use relative pathnames that don't start with a slash -- otherwise, the writing-tar will write the copies in the same place the originals came from!
    "How about an example," you ask? The figure below has one. It copies from the directory /home/jane, with all its files and subdirectories. The copy is made in the directory /work/bkup/jane:
    
% mkdir /work/bkup/jane
% cd /home/jane
% tar cf - . | (cd /work/bkup/jane && tar xBf -)
    The && operator tells the shell to start tar xBf only if the previous command (the cd) succeeded. That prevents tar writing files into the same directory it's reading from -- if the destination directory isn't accessible or you flub its pathname. If your tar has a B (reblocking) option, use it to help be sure that the copy is made correctly. If your tar doesn't have a reblocking option, you can use this trick suggested by Chris Torek:
    
% tar cf - . | cat | (cd /work/backup/jane && tar xbf 1 -)
    You can use other options that your tar might have, like excluding files or directories, on the reading-tar, too. Some gotchas:
    • Symbolic links will be copied exactly. If they point to relative pathnames, the copied links might point to locations that don't exist. You can search for these symbolic links with find - type l.
    • A hard link will be copied as a file. If there are more hard links to that file in the files you're copying, they will be linked to the copy of the first link. That can be good because the destination might be on a different filesystem (a hard link to the original file can't work then). It can be bad if the link pointed to a really big file; the copy can take a lot of disk space. You can search for these hard links by:
      • Searching the directory from which you're copying with find - links +1 - type f to find all files that have more than one link, and
      • Running the reading-tar with its l (lowercase letter L) option to complain if it didn't copy all links to a file.
     
    Posted by at No comments:

    Monday, May 23, 2016

    Hiawatha

    An advanced and secure webserver for Unix

     

    Introduction

    Hiawatha is an open source webserver with a focus on security. I started Hiawatha in January 2002. Before that time, I had used several webservers, but I didn't like them. They had unlogical, almost cryptic configuration syntax and none of them gave me a good feeling about their security and robustness. So, I decided it was time to write my own webserver. I never thought that my webserver would become what it is today, but I enjoyed working on it and liked to have my own open source project. In the years that followed, Hiawatha became a fully functional webserver.

    Secure

    Hiawatha has been written with security in mind. This resulted in a highly secure webserver in both code and features. Hiawatha can stop SQL injections, XSS and CSRF attacks and exploit attempts. Via a specially crafted monitoring tool, you can keep track of all your webservers.

    Easy to use

    You don't need to be a HTTP or CGI expert to get Hiawatha up and running. Its configuration syntax is easy to learn. The documentation and examples you can find on this website will give you all the information you need to configure your webserver within minutes.

    Lightweight

    Although Hiawatha has everything a modern webserver needs, it's nevertheless a small and lightweight webserver. This makes Hiawatha ideal for older hardware or embedded systems. Special techniques are being used to keep the usage of resources as low as possible.

    Official website

    https://www.hiawatha-webserver.org/

    Small but Safe

    12 April 2016, 08:03
    "The small but secure Hiawatha web server provides an appealing alternative to the complex Apache and other alternatives."
    Security on the Internet is vital. The Hiawatha web server is a small (and free) web server that subscribes to the principle "security by default."
    The upcoming edition of the Admin magazine will contain a story about Hiawatha. Many thanks to Hans-Cees Speel.

    by Hugo Leisink



    Posted by at No comments:
    Subscribe to: Posts (Atom)