hakopako

Full-stack Engineer's blog

Useful linux command tips for dealing with log files

日本語 | English

Tips for dealing with text files such as log files when you are required to be done ease aggregation. Usually these depends on machine spec.

Tools

・ Excel: good up to about 50K lines.
・ Sublime text: good up to about 300K lines.
・ For more data, it works fine with linux command. even 10M lines

(Regardless of data amount, I personaly prefer use linux commands)

Check part of data (a few lines)

# show first 10 lines 
$ head -n 10 hoge.log

# see file by filtering a word "piyo". good to see access log.
$ tail -f hoge.log | grep piyo

Aggregate command

# combine some files into one file.
$ cat *.log >> aaa.log

# check the file line count.
$ cat hoge.log | wc -l

# filtering by a word "bar"
$ cat hoge.log | grep bar >> aaa.log

# except line which contains a word "piyo"
$ cat hoge.log | grep -v piyo >> aaa.log

# replace a word aaa -> bbb
$ sed -e s/aaa/bbb/g >> aaa.log

# delete duplicated line without sort in advance
$ awk '!a[$0]++' hoge.log >> aaa.log

 

These commands helps you to do easy aggregation.

Proxy command options and setting file notations - yum/rpm/curl/wget/git/gem/pip/npm -

日本語 | English

If proxy is barproxy:1234.

yum

$ vim /etc/yum.conf
proxy=http://barproxy:1234     # add

rpm

$ rpm --httpproxy barproxy --httpport 1234 http://aaa.com

curl

# --- command --------
$ curl http://aaa.com --proxy barproxy:1234
$ curl -x barproxy:1234 http://aaa.com

# --- configure file ----
$ vim ~./curlrc
proxy="http://barproxy:1234"    # add

wget

# --- command --------
$ wget "http://aaa.com" -e use_proxy=yes -e http_proxy=barproxy:1234
$ wget "https://aaa.com" -e use_proxy=yes -e https_proxy=barproxy:1234

# --- configure file ----
$ vim ~/.wgetrc
http_proxy=http://barproxy:1234    # add
https_proxy=http://barproxy:1234     # add

git

$ git config --global http.proxy http://barproxy:1234
$ git config --global https.proxy http://barproxy:1234

or edit .gitconfig file.

gem

$ gem install aaa --http-proxy http://barproxy:1234
$ gem install aaa -p http://barproxy:1234

rbenv

set wget and curl proxies.

pip

$ pip install aaa --proxy=barproxy:1234

npm

# --- command --------
$ npm --proxy http://barproxy:1234 install aaa
$ npm --https-proxy http://barproxy:1234 install aaa

# --- configure file ----
$ ~/.npmrc
proxy = http://barproxy:1234    # add
https-proxy = http://barproxy:1234    # add

AWS Lambda 101 - how to develop, run, deploy - [nodejs]

日本語 | English

Overview

AWS Lambda 101.
An event of returning s3 file content.
Use node 4.3 for lambda.

Programms

package.json / install aws-sdk
{
  "name": "application-name",
  "version": "1.0.0",
  "description": "application description",
  "main": "application.js",
  "scripts": {
    "test": "node local_run.js"
  },
  "author": "",
  "dependencies": {
    "aws-sdk": "^2.4.13"
  }
}

Don't forget to do $ npm install.

local_run.js / for local running

ex: http://lambda-url/function-name?key=test

'use strict'

// Your data to be passed to lambda function
let event = {
    key: 'test'
}

// Build stab of context Object
let context = {
    functionName: 'function-name',
    invokeid: 'invokeid',
    done: function(err, message) {
        return;
    },
    succeed: function(message) {
        console.log(message)
    }
}

let app = require('./application.js')
app.handler(event, context)
application.js / actual upload file
'use strict';

const AWS = require('aws-sdk')

AWS.config.update({
    accessKeyId: '',
    secretAccessKey: '',
    region: 'ap-northeast-1',
    endpoint: 'https://s3-ap-northeast-1.amazonaws.com'
})

const s3client = new AWS.S3()

exports.handler = (event, context) => {
    let params = {
        Bucket: "hogehoge",
        Key: event.key
    }
    s3client.getObject(params, (err, data) => {
        if (data != null) {
            context.succeed(data.Body.toString())
        }
    })
}

How to run locally

Execute one command below.

$ node local_run.js

Generate zip file and upload it

Generate zip file by an command below.

$ zip -r testfunc.zip application.js node_modules

Then, upload testfunc.zip through AWS web console.
See "console > lambda > target function > Configuration " and set "application.handler" at "Handler" form.