자바 설치 

sudo amazon-linux-extras enable corretto8
sudo yum clean metadata
sudo yum install java-1.8.0-amazon-corretto-devel

자바의 설치 경로를 확인합니다. 

$ which java
/usr/bin/java

$ readlink -f /usr/bin/java
/usr/lib/jvm/java-1.8.0-amazon-corretto.x86_64/jre/bin/java

자바를 환경설정 JAVA_HOME 설정

sudo vi /etc/profile

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-amazon-corretto.x86_64
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar

source /etc/profile

HBase 설치

wget http://archive.apache.org/dist/hbase/1.2.7/hbase-1.2.7-bin.tar.gz
tar -zxvf hbase-1.2.7-bin.tar.gz
rm hbase-1.2.7-bin.tar.gz

hbase-1.2.7/conf/hbase-env.sh 파일 수정

# Configure PermSize. Only needed in JDK7. You can safely remove it for JDK8+
export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m -XX:ReservedCodeCacheSize=256m"
export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m -XX:ReservedCodeCacheSize=256m"

위 두줄을 주석 처리하여 준다. 

hbase-1.2.7/conf/hbase-site.xml 파일 수정

 <configuration>
      <property>
          <name>hbase.rootdir</name>
          <value>file:///home/ec2-user/pinpoint/data</value>
      </property>
      <property>
          <name>hbase.zookeeper.property.dataDir</name>
          <value>/home/ec2-user/pinpoint/zookeeper</value>
      </property>
  </configuration>

실행

hbase-1.2.7/bin/start-hbase.sh

hbase 스키마 생성

cd /home/ec2-user/pinpoint/hbase-1.2.7/bin
wget https://raw.githubusercontent.com/pinpoint-apm/pinpoint/master/hbase/scripts/hbase-create.hbase

TTL 값이 31536000 초(365일), 5184000 초(60 일)로 데이터를 유지하도록 되어있어 604800 초(7일)로 변경

sed 's/31536000/604800/' hbase-create.hbase > hbase-create-real.hbase
sed -i 's/5184000/604800/' hbase-create-real.hbase

스키마 생성

./hbase shell ./hbase-create-real.hbase

Pinpoint Collector 설치 및 실행

wget https://github.com/pinpoint-apm/pinpoint/releases/download/v2.2.0/pinpoint-collector-boot-2.2.0.jar
nohup java -jar -Dpinpoint.zookeeper.address=localhost pinpoint-collector-boot-2.2.0.jar 1>/dev/null 2>&1 &

Pinpoint Web 설치 및 실행

wget https://github.com/pinpoint-apm/pinpoint/releases/download/v2.2.0/pinpoint-web-boot-2.2.0.jar
nohup java -jar -Dpinpoint.zookeeper.address=localhost pinpoint-web-boot-2.2.0.jar 1>/dev/null 2>&1 &

 

Pinpoint Agent 설치

wget https://github.com/pinpoint-apm/pinpoint/releases/download/v2.2.0/pinpoint-agent-2.2.0.tar.gz
tar -zxvf pinpoint-agent-2.2.0.tar.gz

vi /pinpoint-agent-2.2.0/pinpoint-root.config 수정 

profiler.transport.grpc.collector.ip=192.168.0.207

 

아마존 영역에서 elasticsearch를 사용하기 위해서는 opensearch를 사용해야되는데 기존 elasticsearch 와 연동하는 부분이 라이센스 문제 때문에 삽질을 계속했다.  

logstash에서 opensearch로 데이터를 전송하기 위해서는 logstash-oss 다운받아야 하며 opensearch 모듈이 포함되어 있는 것을 다운 받아야 한다. (https://opensearch.org/downloads.html)

curl https://artifacts.opensearch.org/logstash/logstash-oss-with-opensearch-output-plugin-7.16.2-linux-x64.tar.gz -o logstash-oss-with-opensearch-output-plugin-7.16.2-linux-x64.tar.gz

tar -zxvf logstash-oss-with-opensearch-output-plugin-7.16.2-linux-x64.tar.gz

로그스태시 생성

/home/ec2-user/logstash-7.16.2/config/logstash.conf 파일 생성

input {
  file {
    path => ["/log/*.log"]
    start_position => "beginning"
  }
}
filter {
    json {
      source => "message"
      remove_field => ["message", "@timestamp", "@version"]
    }
}
output {
     opensearch {
        hosts       => ["https://:443"]
        user        => "user"
        password    => "password"
        index       => "logstash-logs"
    }
}

로그스태시  실행 

/home/ec2-user/logstash-7.16.2/bin/logstash -f /home/ec2-user/logstash-7.16.2/config/test.conf

토픽 생성

bin/kafka-topics.sh --create \
--bootstrap-server 카프카서버:9092 \
--replication-factor 1 \
--partitions 3 \
--topic 토픽이름

토픽 삭제

bin/kafka-topics.sh --delete --bootstrap-server 카프카서버:2181 --topic 토픽이름

토픽 리스트

bin/kafka-topics.sh  --bootstrap-server 카프카서버:9092 --list

토픽 상세

bin/kafka-topics.sh --bootstrap-server 카프카서버:9092 --describe --topic 토픽이름

컨슈머

컨슈머 확인

bin/kafka-console-consumer.sh --bootstrap-server 카프카서버:9092 --topic 토픽이름 --from-beginning

컨슈머 그룹 리스트

bin/kafka-consumer-groups.sh --bootstrap-server 카프카서버:9092 --list

컨슈머 그룹 상세

bin/kafka-consumer-groups.sh --bootstrap-server 카프카서버:9092 --describe --group 컨슈머그룹이름

+ Recent posts

티스토리툴바