[ask] Cara setting robots.txt?

Discussion in 'General Internet' started by ziuma, Oct 24, 2017.

  1. ziuma

    ziuma Well-Known Member

    Joined:
    May 23, 2014
    Messages:
    1,563
    Likes Received:
    240
    Trophy Points:
    63
    Selamat Malam semua,

    Saya mau tanya sedikit, mudah-mudahan dapat jawaban yang banyak :D
    bagaimana ya cara setting robots.txt

    supaya robots.txt tersembunyi dari pengunjung tetapi bot dari search engine masih bisa membacanya

    maksud saya: robots.txt hanya untuk bot search engine aja

    Terima Kasih untuk bantuannya
     
  2. Rimala Nursery

    Rimala Nursery Super Level

    Joined:
    May 29, 2014
    Messages:
    2,089
    Likes Received:
    254
    Trophy Points:
    83
    Saya malah gak pernah seting karena gak tau caranya *boohoo* kalau salah seting bisa berpengaruh katanya ,begitupun kalo benar.. haha
     
  3. rosyad

    rosyad Active Member

    Joined:
    Dec 30, 2016
    Messages:
    1,106
    Likes Received:
    84
    Trophy Points:
    48
  4. ys. herbi

    ys. herbi Well-Known Member

    Joined:
    Mar 6, 2016
    Messages:
    1,251
    Likes Received:
    190
    Trophy Points:
    63
    Google+:
    Setting robot.txt.. saya pernah sih, tp ndak tau bagusnya gimana...
    Soalnya dulu pernah salah nyebut, ndak diindeks SE tertentu (lupa namanya)...

    Kok bisa tau mas?
    Lha abis nglepas robot.txt itu, ada SE "asing" yang banyak ngerujuk ke blog saya..
     
  5. KangAndre

    KangAndre Member

    Joined:
    Jan 25, 2014
    Messages:
    10,244
    Likes Received:
    2,714
    Trophy Points:
    413
    jangan buat atau hapus robot.txt, robot akan tetap meng-crawl

    Robot.txt diperlukan sebagai aturan bagi robot biar gak main selonong saja, namun ada juga robot yang "nakal" meskipun ada robot.txt tetap main serobot.
     
  6. ziuma

    ziuma Well-Known Member

    Joined:
    May 23, 2014
    Messages:
    1,563
    Likes Received:
    240
    Trophy Points:
    63
    ngak bisa di hide ya @KangAndre ?
    artinya robots.txt nya ada tetapi pengunjung tidak bisa lihat

    saya pasang kode seperti ini di .hattacces

    Code:
    <Files robots.txt>
    order allow,deny
    deny from all
    </Files>
    
    ternyata di google webmaster dianggap tidak ada robots.txt
     
  7. KangAndre

    KangAndre Member

    Joined:
    Jan 25, 2014
    Messages:
    10,244
    Likes Received:
    2,714
    Trophy Points:
    413
    Tidak disarankan membuat batasan robots di .htaccess.
    Alasannya:
    .htaccess digunakan untuk akses internal , sedangkan robots.txt untuk akses eksternal .

    "Internal" karena .htaccess bertugas untuk memberitahu server (Apache) bagaimana menangani nama halaman dan file, URL dan cara pengguna untuk mengakses sumber daya ini( untuk menangani fitur-fitur situs sendiri). Salah pengisian situs bisa error.

    Sedangkan Robots.txt, sebaliknya mengatur akses "eksternal", karena ini memberi tahu mesin pencari dan alat web lainnya yang harus dibaca dan yang boleh diindeks. Jika salah, hanya berakibat tidak terindeks saja, tapi situs tetap berjalan.

    Sudah ada ketentuan bahwa robots.tst dapat dibaca manusia (pengguna) dan robot.

    ziuma.com/robots.txt terbaca, coba hapus yang di .htaccess dan uji kembali via webmaster tools.
     
    ziuma likes this.
  8. davidrahman

    davidrahman Member

    Joined:
    Jun 4, 2014
    Messages:
    899
    Likes Received:
    32
    Trophy Points:
    28
    sepertinya ada hal yang penting nih di robots.txt punya ts :D
     
  9. KangAndre

    KangAndre Member

    Joined:
    Jan 25, 2014
    Messages:
    10,244
    Likes Received:
    2,714
    Trophy Points:
    413
    Semua situs bisa dilihat robots.txt-nya
    namadomain.com/robots.txt dan nggak ada rahasia penting :D
     
  10. ncang

    ncang Super Level

    Joined:
    Feb 7, 2013
    Messages:
    4,655
    Likes Received:
    761
    Trophy Points:
    113
    Google+:
    ada yang keren juga isinya,
    Code:
    https://www.nike.com/robots.txt
    https://www.google.com/killer-robots.txt
    
    ini barusan screenshot, gak pake :D
    [​IMG]
     
    ziuma likes this.
  11. ziuma

    ziuma Well-Known Member

    Joined:
    May 23, 2014
    Messages:
    1,563
    Likes Received:
    240
    Trophy Points:
    63
    ada logo nike dibawahnya
    keren keren :D
     
  12. KangAndre

    KangAndre Member

    Joined:
    Jan 25, 2014
    Messages:
    10,244
    Likes Received:
    2,714
    Trophy Points:
    413
    Tapi jarang orang lihat robots.txt (paling iseng saja) :D
     
  13. Sugeng Ginanjar

    Sugeng Ginanjar Member

    Joined:
    Nov 18, 2016
    Messages:
    213
    Likes Received:
    10
    Trophy Points:
    18
  14. Andre

    Andre Member

    Joined:
    Apr 12, 2013
    Messages:
    483
    Likes Received:
    22
    Trophy Points:
    18
  15. miftah666

    miftah666 Member

    Joined:
    Sep 6, 2016
    Messages:
    22
    Likes Received:
    0
    Trophy Points:
    8
    Mendingan jangan di hapus Gan.....menurut saya
     
  16. Fathoni

    Fathoni Member

    Joined:
    Mar 9, 2016
    Messages:
    168
    Likes Received:
    12
    Trophy Points:
    18
    ikutan nyimak gan biar semakin pintar dan bertambah ilmu lagi
    :D
     
  17. Nina Ayunine

    Nina Ayunine Member

    Joined:
    Nov 2, 2017
    Messages:
    27
    Likes Received:
    4
    Trophy Points:
    8
    tanya bawah ane mungkin paham
     
  18. Fathoni

    Fathoni Member

    Joined:
    Mar 9, 2016
    Messages:
    168
    Likes Received:
    12
    Trophy Points:
    18
  19. Nastriawan

    Nastriawan Member

    Joined:
    Sep 7, 2016
    Messages:
    60
    Likes Received:
    6
    Trophy Points:
    8
    ijin nyimak gan, sy juga msh belum ngerti
     
Loading...

Share This Page